1-gentle-introductory-series/volume-2-basic-operations-on-numeric-tables.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#  Gentle Introduction to PyDAAL: Vol 2 of 3 Basic Operations on Numeric Tables"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A wide range of classes are available in the Intel® Data Analytics Acceleration Library (Intel® DAAL) to create a numeric table accommodating various data layout, dtypes, and frequent access methods. [Volume 1](volume-1-data-structures.ipynb) of this series covers numeric table creation under different scenarios. Once created, Intel® DAAL provides operational methods for visualizing and mutating a user’s numeric tables. Volume 2 will cover the usage of the operational methods. Subsequently Volume 3 in this series gives a brief introduction to Algorithm section of PyDaal. [Table 1](#Table1) can be used as a quick reference for basic operations on Intel® DAAL’s numeric table\n",
    "\n",
    "## Volumes in Gentle Introduction Series\n",
    "\n",
    "•\t**[Vol 1: Data Structures](volume-1-data-structures.ipynb)** - Covers introduction to  Data Management component of Intel® DAAL and available custom Data Structures(Numeric Table and Data Dictionary) with code examples.<br>\n",
    "•\t**Vol 2: Basic Operations on Numeric Tables** - Covers introduction to possible operations that can be performed on Intel® DAAL's custom Data Structure (Numeric Table and Data Dictionary) with code examples.<br>\n",
    "•\t**[Vol 3: Analytics Model Building and Deployment](volume-3-analytics-model-building-deployment.ipynb)** – Covers introduction to analytics model building and evaluation in Intel® DAAL with serialized deployment and distributed model fitting on large datasets.<br>\n",
    "\n",
    "###  <a id='Table1'> *Table 1. Quick reference table on available methods*</a>\n",
    "\n",
    "|Method Description|Usage Syntax|\n",
    "|:-|:-|\n",
    "|*Print numeric table as stored in memory to represent data layout. <br> Method requires ‘nT’ as input argument*| printNumericTable(nT)|\n",
    "|*Quick visualization on multiple numeric tables*|printNumericTables(nT1,nT2)|\n",
    "|Check shape of numeric table|*#Number of Rows*<br>nT.getNumberOfRows()<br><br>*#Number of Columns*<br>nT.getNumberOfColumns()|\n",
    "|Allocate buffer to load block of numeric table for access and manipulation operations.| block = BlockDescriptor_Float64()<br>*#Allocates a memory block with double dtype*|\n",
    "|Retrieve block of rows and columns from numeric table into Block Descriptor for visualization.<br>(Setting rwflag  to ‘readOnly’ enables only read access to the buffer.)|*#Block of Column values*<br> nT.getBlockOfColumnValues(colIndex, firstRowIndex,lastRowIndex, rwflag, block)<br><br>*#Block of Rows*<br> nT.getBlockOfRows(firstRowIndex,lastRowIndex,  rwflag, block)\n",
    "|Extract numpy array  from Block Descriptor object when loaded with block of values|block.getArray()|\n",
    "|Release block of Rows from buffer|nT.releaseBlockOfRows(block)|\n",
    "|*Print underlying array of numeric table. Method requires ‘np.array’ as input argument*|printArrayblock.getArray),num_printed_cols, num_printed_rows, num_cols, message)|\n",
    "|Check FeatureTypes on each column of numeric table data dictionary|dict[colIndex].featureType|\n",
    "\n",
    "\n",
    "*denotes functions included in the ‘utils’ folder, which can be found in \"/share/pydaal_examples/examples/python/source/\".\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "##  Different phases of life Cycle"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## 1.\tInitiate\n",
    "\n",
    "Let’s begin by constructing a numeric table (nT) directly from a Numpy array. We will use the nT throughout the rest of the code examples in this volume."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from daal.data_management import(FileDataSource, DataSource)\n",
    "\n",
    "dataSource = FileDataSource(\n",
    "    r'_notebook-related-files\\normalization.csv', DataSource.doAllocateNumericTable, DataSource.doDictionaryFromContext)\n",
    "\n",
    "nT = HomogenNumericTable()\n",
    "dataSource.loadDataBlock(nT)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.\tOperate\n",
    "\n",
    "Once initialized, numeric tables provide various classes and member functions to access and manipulate data similar to a pandas DataFrame. We will dive next into Intel DAAL’s operational methods, after an important note about Intel DAAL’s bookkeeping object called Data Dictionary.\n",
    "\n",
    "### Data Dictionary:\n",
    "\n",
    "As mentioned in Volume 1 of this series on creation of Intel DAAL’s numeric tables , these custom data structures must be accompanied by a Data Dictionary to perform operations. When raw data streams into memory to populate the numeric table structure, the table’s Data Dictionary concurrently records metadata. Dictionary creation will occur automatically unless specified to not allocate by the user. Various Data Dictionary methods are available to access and manipulate feature type, dtypes etc. If a user creates a numeric table without memory allocation, the Data Dictionary values have to be explicitly set with feature types. An important note is that Intel DAAL’s Data Dictionary is a custom data structure, not a Python dictionary.\n",
    "\n",
    "\n",
    "More details on working with Intel DAAL [Data Dictionaries](#ChangeFeatureType)\n",
    "\n",
    "## 2.1\tData Mutation in Numeric Table:\n",
    "   ### 2.1.1\tStandardization and Normalization:\n",
    "   \n",
    "Data analysis work is usually preceded by a Data Preprocessing stage that includes data wrangling, quality checks, and assurance to handle null values, outliers etc. An important preprocessing activity is to normalize input data. Intel DAAL offers routines to support two popular normalization techniques on numeric tables: Namely, Z-score standardization and Min-Max normalization.<br>\n",
    "\n",
    "*NOTE: Currently, Intel DAAL only supports rescaling for descriptive analytics. In the future, support will be added for predictive analytics with the addition of a “transform()” method to be applied to new data.*\n",
    "\n",
    "* **Z-score Standardization:**\n",
    "\n",
    "Rescales numeric table values feature-wise to the number of standard deviations each value deviates from the mean. Below are the steps to use Intel DAAL’s z-score standardization.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import daal.algorithms.normalization.zscore as zscore\n",
    "\n",
    "# Create an algorithm\n",
    "algorithm = zscore.Batch(method=zscore.sumDense)\n",
    "\n",
    "# Set input object for the algorithm to nT\n",
    "algorithm.input.set(zscore.data, nT)\n",
    "\n",
    "# Compute Z-score normalization function\n",
    "res = algorithm.compute()\n",
    "\n",
    "#Retrieve normalized nT\n",
    "Norm_nT= res.get(zscore.normalizedData)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* **Min-Max Normalization:**\n",
    "\n",
    "Rescales numeric table values feature-wise linearly to fit [0, 1] / [-1-1] range. Below are the steps to use Intel DAAL’s Min-Max normalization.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import daal.algorithms.normalization.minmax as minmax\n",
    "\n",
    "# Create an algorithm\n",
    "algorithm = minmax.Batch(method=minmax.defaultDense)\n",
    "\n",
    "# Set lower and upper bounds for the algorithm\n",
    "algorithm.parameter.lowerBound = -1.0\n",
    "algorithm.parameter.upperBound = 1.0\n",
    "\n",
    "# Set input object for the algorithm to nT\n",
    "algorithm.input.set(minmax.data, nT)\n",
    "\n",
    "# Compute Min-max normalization function\n",
    "res = algorithm.compute()\n",
    "\n",
    "# Get normalized numeric table\n",
    "Norm_nT = res.get(minmax.normalizedData)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.1.2\tBlock Descriptor for Visualization and Mutation: \n",
    "\n",
    "The Contents of a numeric table cannot be accessed directly to visualize or manipulate. Instead a user must first move a requested block of data values to a memory buffer. Once instantiated, this buffer is housed in an object called BlockDescriptor. An Intel DAAL numeric table object has member functions to retrieve blocks of rows/columns and add to the BlockDescriptor. The argument rwflag is used to set “readOnly”/“readWrite” mode, depending on whether the user intends to update values in the numeric table while releasing the block. Conveniently, the BlockDescriptor class allows for block retrieval of data in specific rows and/or columns.<br> \n",
    "\n",
    "*NOTE: the dtype of data in the BlockDescriptor buffer is not required to match the numeric table that sourced the block.*\n",
    "\n",
    "**Access Modes:**\n",
    "\n",
    "* “readOnly” argument sets rwflag to provide read only access to numeric table contents, thus performing no updates to the table when the block is released from buffer memory.<br><br>\n",
    "***Syntax and Usage:***\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from daal.data_management import BlockDescriptor_Float64, readOnly\n",
    "#Allocate a readOnly memory block with double dtype \n",
    "block = BlockDescriptor_Float64()\n",
    "nT.getBlockOfRows(0,1, readOnly, block)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* “readWrite” argument sets rwflag to write back any changes from block descriptor object to the numeric table when the block is released from buffer memory, thus enabling numeric table mutation with the help of block descriptor.<br><br>\n",
    "***Syntax and Usage:***\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from daal.data_management import BlockDescriptor_Float64, readWrite\n",
    "#Allocate a readOnly memory block with double dtype \n",
    "block = BlockDescriptor_Float64()\n",
    "nT.getBlockOfRows(0,1, readWrite, block)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.1.3\tBlockDescriptor() in “readWrite” mode: \n",
    "\n",
    "When rwflag argument is set to “readWrite” in getBlockOfRows() getBlockOfColumnValues(), contents of BlockDecriptor object are written back to the numeric table while releasing block of rows, making edits possible on existing rows/columns in numeric table.<br>\n",
    "\n",
    "Let’s create a basic numeric table to explain BlockDecriptor in “readWrite” mode in detail\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from daal.data_management import HomogenNumericTable, readWrite, BlockDescriptor\n",
    "from CustomUtils import getArrayFromNT\n",
    "array =np.array([[1,2,3,4],\n",
    "                [5,6,7,8]])\n",
    "nT= HomogenNumericTable(array)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* **Edit numeric table Row-wise:**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Create buffer object with ntype \"double\"\n",
    "doubleBlock = BlockDescriptor(ntype=np.float64)\n",
    "\n",
    "firstRow = 0\n",
    "lastRow = nT.getNumberOfRows()\n",
    "\n",
    "#getBlockOfRows() member function in \"readWrite\" mode to retrieve numeric table contents and populate \"doubleBlock\" object\n",
    "nT.getBlockOfRows(firstRow,lastRow, readWrite, doubleBlock)\n",
    "#Access array contents from \"doubleBlock\" object\n",
    "array = doubleBlock.getArray()\n",
    "#Mutate 1st row of array to reflect on \"doubleBlock\" object\n",
    "array[0] = [0,0,0,0]\n",
    "#Release buffer object and write changes back to numeric table\n",
    "nT.releaseBlockOfRows(doubleBlock)\n",
    "print(getArrayFromNT(nT))\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Note: nT was originally created with data [[1,2,3,4],[5,6,7,8]]. After row mutation the first row is now replaced using buffer memory. Updated nT has data [[0,0,0,0\\],[5,6,7,8]].*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "* **Edit numeric table Column-wise:**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Create  buffer object with ntype \"double\"\n",
    "doubleBlock = BlockDescriptor(ntype=np.intc)\n",
    "ColIndex = 2\n",
    "firstRow = 0\n",
    "lastRow = nT.getNumberOfRows()\n",
    "\n",
    "#getBlockOfColumnValues() member function in \"readWrite\" mode to retrieve numeric table ColIndex contents and populate \"doubleBlock\" object\n",
    "nT.getBlockOfColumnValues(ColIndex,firstRow,lastRow,readWrite,doubleBlock)\n",
    "\n",
    "#Access array contents from \"doubleBlock\" object\n",
    "array = doubleBlock.getArray()\n",
    "\n",
    "#Mutate array to reflect on \"doubleBlock\" object\n",
    "array[:][:] = 0\n",
    "\n",
    "#Release buffer object and write changes back to numeric table\n",
    "nT.releaseBlockOfColumnValues(doubleBlock)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Note: nT was originally created with data  [[1,2,3,4],[5,6,7,8]] After column mutations, the third column is replaced with [0,0] using buffer memory. Updated nT has data [[1,2,0,4],[5,6,0,8]].*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.1.4\tMerge numeric table:\n",
    "\n",
    "Numeric tables can be appended along rows and columns, provided, they share the same array size along the relevant axis to merge. \n",
    "\n",
    "RowMergedNumericTable()and MergedNumericTable() are the 2 classes available to merge numeric tables. The latter is used for merges on column indexes.\n",
    "\n",
    "* **Merge Row-wise: **\n",
    "\n",
    "***Syntax:***\n",
    "\n",
    "mnT = RowMergedNumericTable()\n",
    "\n",
    "mnT.addNumericTable(nT1); mnT.addNumericTable(nT2); mnt.addNumericTable(nT3)\n",
    "\n",
    "***Code Example:***\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from daal.data_management import HomogenNumericTable, RowMergedNumericTable\n",
    "import numpy as np\n",
    "from utils import printNumericTable\n",
    "\n",
    "#nT1 and nT2 are 2 numeric tables having equal number of COLUMNS\n",
    "array =np.array([[1,2,3,4],\n",
    "                 [5,6,7,8]], dtype = np.intc)\n",
    "nT1= HomogenNumericTable(array)\n",
    "array =np.array([[9,10,11,12],\n",
    "                 [13,14,15,16]],dtype = np.intc)\n",
    "nT2= HomogenNumericTable(array)\n",
    "\n",
    "#Create merge numeric table object\n",
    "mnT = RowMergedNumericTable()\n",
    "\n",
    "#add numeric tables to merged numeric table object\n",
    "mnT.addNumericTable(nT1); mnT.addNumericTable(nT2)\n",
    "printNumericTable(mnT)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* **Merge Column-wise:**\n",
    "\n",
    "***Syntax:***\n",
    "\n",
    "\n",
    "mnT = MergedNumericTable()\n",
    "\n",
    "mnT.addNumericTable(nT1); mnT.addNumericTable(nT1); mnt.addNumericTable(nT3)</font>\n",
    "\n",
    "***Code Example:***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "\n",
    "from daal.data_management import HomogenNumericTable, MergedNumericTable\n",
    "import numpy as np\n",
    "from utils import printNumericTable\n",
    "\n",
    "#nT1 and nT2 are 2 numeric tables having equal number of ROWS\n",
    "array =np.array([[1,2,3,4],\n",
    "                 [5,6,7,8]], dtype = np.intc)\n",
    "nT1= HomogenNumericTable(array)\n",
    "array =np.array([[9,10,11,12],\n",
    "                 [13,14,15,16]],dtype = np.intc)\n",
    "nT2= HomogenNumericTable(array)\n",
    "\n",
    "#Create merge numeric table object\n",
    "mnT = MergedNumericTable()\n",
    "\n",
    "#add numeric tables to merged numeric table object\n",
    "mnT.addNumericTable(nT1); mnT.addNumericTable(nT2)\n",
    "printNumericTable(mnT)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### <font size=\"+1\"> 2.1.5 Split Numeric table:</font>\n",
    "\n",
    "See [Table 1](#Table1) for a quick reference on available methods for the entries getBlockOfRows()and  getBlockOfColumnValues() methods, used to extract sections of a numeric table by row or column values. Additionally, the helper function getBlockOfNumericTable() is provided below and implements the capability to extract a contiguous subset of the table with selected range of rows and columns. getBlockOfNumericTable()accepts int or list keyword arguments for ranges of rows and columns, using conventional Python 0 - based indexing.\n",
    "\n",
    "***Syntax:***\n",
    "getBlockOfNumericTable(nT, Rows = ‘All’, Columns = ‘All’)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***Helper Function:***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def getBlockOfNumericTable(nT,Rows = 'All', Columns = 'All'):\n",
    "    from daal.data_management import HomogenNumericTable_Float64, \\\n",
    "    MergedNumericTable, readOnly, BlockDescriptor\n",
    "    import numpy as np\n",
    "#------------------------------------------------------\n",
    "    # Get First and Last Row indexes\n",
    "    lastRow = nT.getNumberOfRows()\n",
    "    if type(Rows)!= str:\n",
    "        if type(Rows) == list:\n",
    "            firstRow = Rows[0]\n",
    "            if len(Rows) == 2: lastRow = min(Rows[1], lastRow)\n",
    "        else:firstRow = 0; lastRow = Rows\n",
    "    elif Rows== 'All':firstRow = 0\n",
    "    else:\n",
    "        warnings.warn('Type error in \"Rows\" arguments, Can be only int/list type')\n",
    "        raise SystemExit\n",
    "#------------------------------------------------------\n",
    "    # Get First and Last Column indexes\n",
    "    nEndDim = nT.getNumberOfColumns()\n",
    "    if type(Columns)!= str:\n",
    "        if type(Columns) == list:\n",
    "            nStartDim = Columns[0]\n",
    "            if len(Columns) == 2: nEndDim = min(Columns[1], nEndDim)\n",
    "        else: nStartDim = 0; nEndDim = Columns\n",
    "    elif Columns == 'All': nStartDim = 0\n",
    "    else:\n",
    "        warnings.warn ('Type error in \"Columns\" arguments, Can be only int/list type')\n",
    "        raise SystemExit\n",
    "#------------------------------------------------------\n",
    "    #Retrieve block of Columns Values within First & Last Rows\n",
    "    #Merge all the retrieved block of Columns Values\n",
    "    #Return merged numeric table\n",
    "    mnT = MergedNumericTable()\n",
    "    for idx in range(nStartDim,nEndDim):\n",
    "        block = BlockDescriptor()\n",
    "        nT.getBlockOfColumnValues(idx,firstRow,(lastRow-firstRow),readOnly,block)\n",
    "        mnT.addNumericTable(HomogenNumericTable_Float64(block.getArray()))\n",
    "        nT.releaseBlockOfColumnValues(block)\n",
    "    block = BlockDescriptor()\n",
    "    mnT.getBlockOfRows (0, mnT.getNumberOfRows(), readOnly, block)\n",
    "    mnT = HomogenNumericTable (block.getArray())\n",
    "    return mnT\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***Usage:***\n",
    "\n",
    "There are 4 different ways of passing arguments to this function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "getBlockOfNumericTable(nT)\n",
    "#Extracts block of numeric table having all rows and columns of nT. \n",
    "getBlockOfNumericTable(nT, Rows = 4, Columns = 5) \n",
    "# Retrieves first 4 rows and first 5 column values of nT \n",
    "getBlockOfNumericTable(nT, Rows=[2,4], Columns = [1,3]) \n",
    "# Slices numeric table along row and column directions using lower bound and upper bound passed as parameters in list\n",
    "getBlockOfNumericTable(nT, Rows=[1,], Columns = [1,])\n",
    "# Extracts all rows and columns from lower bound through last index."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <a id='ChangeFeatureType'>2.1.6\tChange feature type:</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Numeric table objects have dictionary manipulation methods to get and set feature types in the Data Dictionary for each column. Categorical(0), Ordinal(1), and Continuous(2) are available feature types in Data Dictionary supported by Intel DAAL.\n",
    "\n",
    "* **Get dictionary object associated with nT : **\n",
    "\n",
    "***Syntax:***\n",
    "\n",
    "nT.getDictionary()\n",
    "\n",
    "***Code Example:***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from daal.data_management import data_feature_utils, NumericTableDictionary\n",
    "\n",
    "dict = nT.getDictionary() # nT is numeric table created in section 1\n",
    "'''\n",
    "'dict' object has data dictionary of numeric table nT. This can be used to update metadata information about the data. Most common use case is to modify default feature type of nT columns.\n",
    "'''\n",
    "# Print default feature type of 3rd feature (example feature is continuous):\n",
    "print(dict[2].featureType) #outputs “2” (denotes Continuous feature type) \n",
    "\n",
    "# Modify feature type from Continuous to Categorical:\n",
    "dict[2].featureType = data_feature_utils.DAAL_CATEGORICAL \n",
    "print(dict[2].featureType) #outputs “0” (denotes Categorical feature type)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* **Set dictionary object associated with nT:**\n",
    "\n",
    "This is the method used to replace current Data Dictionary values or to create new Data Dictionaries, if needed. Also, for batch updates, an existing Data Dictionary can be overwritten in full using setDictionary() method.\n",
    "\n",
    "When tables are created without allocating memory for the Data Dictionary, the setDictionary() method must be used to construct metadata for features in the table. Let us again consider nT created in section-1 having 4 features\n",
    "\n",
    "\n",
    "***Syntax:***\n",
    "nT.setDictionary()\n",
    "\n",
    "***Code Example:***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "nT.releaseBlockOfRows(block)\n",
    "\n",
    "nFeatures = 4\n",
    "#Create a dictionary object using Numeric table dictionary class with the number of features\n",
    "dict = NumericTableDictionary(nFeatures) \n",
    "#Allocate a feature type for each feature\n",
    "dict[0].featureType = data_feature_utils.DAAL_CONTINUOUS\n",
    "dict[1].featureType = data_feature_utils.DAAL_CATEGORICAL\n",
    "dict[2].featureType = data_feature_utils.DAAL_CONTINUOUS\n",
    "dict[3].featureType = data_feature_utils.DAAL_CATEGORICAL\n",
    "\n",
    "#set the nT numeric table dictionary with “dict”\n",
    "nT.setDictionary(dict)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.2\tExport Numeric Table to disk:\n",
    "\n",
    "Numeric tables can be exported and saved as a numpy binary file (.npy) file to disk. The following two sections contain helper functions to complete the task of saving in binary form, as well as compressing the data on disk.\n",
    "\n",
    "   #### 2.2.1\tSerialization: \n",
    "   \n",
    "Intel DAAL provides interfaces to serialize numeric table objects into a data archive that can be converted to a numpy array object. The resulting Numpy array, which houses the serialized form of the data, can be saved to disk and subsequently reloaded in the future to reconstruct the source numeric table.<br>\n",
    "\n",
    "To automate this process, the following helper function can be used to serialize and save to disk.<br>\n",
    "\n",
    "\n",
    "***Helper Function:***\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def Serialize(nT):\n",
    "#Construct input data archive Object\n",
    "#Serialize nT contents into data archive Object\n",
    "#Copy data archive contents to numpy array\n",
    "#Save numpy array as .npy in the path\n",
    "    from daal.data_management import InputDataArchive\n",
    "    import numpy as np\n",
    "    \n",
    "    dataArch = InputDataArchive()\n",
    "\n",
    "    nT.serialize(dataArch)\n",
    "\n",
    "    length = dataArch.getSizeOfArchive()\n",
    "    buffer_array = np.zeros(length, dtype=np.ubyte)\n",
    "    dataArch.copyArchiveToArray(buffer_array)\n",
    "\n",
    "    return buffer_array\n",
    "buffer_array = Serialize(nT) # call helper function\n",
    "#np.save(<path>, buffer)# This step is optional\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2.2.2\tCompression:\n",
    "\n",
    "Compressor methods are also available in Intel DAAL to achieve reduced memory footprint when large datasets must be stored to disk. A serialized array representation of an Intel DAAL numeric table can be compressed before saving it to disk, hence achieving optimal storage.<br>\n",
    "\n",
    "To automate this process, the following helper function can be used to serialize, then compress the resulting serialized array.<br>\n",
    "\n",
    "Incorporate helper functions Serialize(nT and CompressToDisk (nT, path) to compress and write numeric tables to disk.\n",
    "\n",
    "***Helper Function:***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def CompressToDisk(nT, path):\n",
    "    # Serialize nT contents\n",
    "    # Create a compressor object\n",
    "    # Create a stream for compression\n",
    "    # Write numeric table to the compression stream\n",
    "    # Allocate memory to store the compressed data\n",
    "    # Store compressed data\n",
    "    # Save compressed data to disk\n",
    "    from daal.data_management import Compressor_Zlib, level9, CompressionStream\n",
    "    import numpy as np\n",
    "\n",
    "    buffer = Serialize (nT)\n",
    "    compressor = Compressor_Zlib ()\n",
    "    compressor.parameter.gzHeader = True\n",
    "    compressor.parameter.level = level9\n",
    "    comprStream = CompressionStream (compressor)\n",
    "    comprStream.push_back (buffer)\n",
    "    compressedData = np.empty (comprStream.getCompressedDataSize (), dtype=np.uint8)\n",
    "    comprStream.copyCompressedArray (compressedData)\n",
    "    np.save (path, compressedData)\n",
    "    CompressToDisk (nT, < path >)# <path>: Give the path name"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.3\tImport Numeric Table from disk:\n",
    "As mentioned in the previous sections, numeric tables can be stored in the form of either serialized or compressed numpy files. Decompression/ Deserialization methods are available to reconstruct the numeric table."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " #### 2.3.1\tDeserialization:\n",
    "\n",
    " The helper function below is available to reconstruct a numeric table from serialized array objects. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def DeSerialize(buffer_array):\n",
    "    from daal.data_management import OutputDataArchive, HomogenNumericTable\n",
    "    #Load serialized contents to construct output data archive object\n",
    "    #De-serialize into nT object and return nT\n",
    "\n",
    "    dataArch = OutputDataArchive(buffer_array)\n",
    "    nT = HomogenNumericTable()\n",
    "    nT.deserialize(dataArch)\n",
    "    return nT\n",
    "#buffer_array = np.load(path) # this step is optional, used only when serialized contents have to be written to  disk\n",
    "nT = DeSerialize(buffer_array)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " #### 2.3.2 Decompression: \n",
    "\n",
    "As compression stage involves serialization of numeric table object, decompression stage includes deserialization. See DeSerialize helper function to recover the numeric table. Refer below for a quick de-compression helper function.<br>\n",
    "\n",
    "Incorporate helper functions DeSerialize(buffer_array) and DeCompressFromDisk(path) to compress and read numeric tables from disk.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def DeCompressFromDisk(path):\n",
    "    from daal.data_management import  Decompressor_Zlib, DecompressionStream\n",
    "    # Create a decompressor\n",
    "    decompressor = Decompressor_Zlib()\n",
    "    decompressor.parameter.gzHeader = True \n",
    "\n",
    "    # Create a stream for decompression\n",
    "    deComprStream = DecompressionStream(decompressor)\n",
    "\n",
    "    # Write the compressed data to the decompression stream and decompress it\n",
    "    deComprStream.push_back(np.load(path))\n",
    "\n",
    "    # Allocate memory to store the decompressed data\n",
    "    deCompressedData = np.empty(deComprStream.getDecompressedDataSize(), dtype=np.uint8)\n",
    "\n",
    "    # Store the decompressed data\n",
    "    deComprStream.copyDecompressedArray(deCompressedData)\n",
    "\n",
    "    #Deserialize\n",
    "    return DeSerialize(deCompressedData)\n",
    "\n",
    "nT = DeCompressFromDisk(<path>)#path must be ‘.npy’ file\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Note: Intel DAAL also implements several other generic compression and decompression methods that include ZLIB, LZO, RLE, and BZIP([reference](https://software.intel.com/sites/products/documentation/doclib/daal/daal-user-and-reference-guides/index.html))*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusion:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Intel® DAAL’s data management component provides classes and methods to perform common operations on numeric table contents. Some of the basic numeric table operations include - access, mutation, export to disk and import from disk. Helper functions covered in this document will help automating Intel® DAAL’s creation of numeric table subsets, as well as serialization and compression processes.\n",
    "\n",
    "The next volume ([Volume 3](volume-3-analytics-model-building-deployment.ipynb)) in the Gentle Introduction series gives a brief introduction to Algorithm section of PyDAAL. Volume 3 focuses on the workflow of important descriptive and predictive algorithms available in Intel® DAAL. Advanced features such as setting hyperparameters, distributing fit calculations, and deploying models as serialized objects will all be covered. \n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##  Other Related Links:\n",
    "\n",
    "•\t[Gentle Introduction to PyDAAL: Vol 1 of 3 Data Structures](volume-1-data-structures.ipynb)<br>\n",
    "•\t[Gentle Introduction to PyDAAL: Vol 3 of 3 Basic Operations on Numeric Tables](volume-3-analytics-model-building-deployment.ipynb)<br>\n",
    "•\t[Developer Guide for Intel® DAAL](https://software.intel.com/en-us/daal-programming-guide)<br>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}