Python_ReadingData.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Reading Data\n",
    "\n",
    "<img style=\"float: right;\" src=\"./images/DataReading.png\" width=\"120\"/>\n",
    "\n",
    "* Astropy has a large number of different ways to read data from external files. \n",
    "* Astropy supports almost any type of file you can think of, from simple text files to complex binary formats.\n",
    "* Most of our datafiles will be `csv` files (comma separated values)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from astropy.table import QTable"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Let us read-in the file: `./Data/StarData.csv`\n",
    "\n",
    "```\n",
    "StarID,Parallax,GMag\n",
    "A5853,768.07,8.98\n",
    "B4472,546.98,8.19\n",
    "C3864,415.18,11.03\n",
    "D7628,392.75,6.55\n",
    "```\n",
    "\n",
    "### `ascii.csv` assumes the first row is a list of the column names"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "star_table = QTable.read('./Data/StarData.csv', \n",
    "                         format='ascii.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "star_table"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "----\n",
    "\n",
    "# Reading (bad) Data\n",
    "\n",
    "<img style=\"float: right;\" src=\"./images/Lore.jpg\" width=\"200\"/>\n",
    "\n",
    "## Different Delimiters\n",
    "\n",
    "Some people just want to watch the world burn, so they create datasets where the columns are separted by something other than a comma."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Bad - Using another delimiter like `:`\n",
    "\n",
    "##### `./Data/StarData_Ver2.dat`\n",
    "\n",
    "```\n",
    "StarID:Parallax:GMag\n",
    "A5853:768.07:8.98\n",
    "B4472:546.98:8.19\n",
    "C3864:415.18:11.03\n",
    "D7628:392.75:6.55\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "star_table_2 = QTable.read('./Data/StarData_Ver2.dat', \n",
    "                           format='ascii.csv',\n",
    "                           delimiter = \":\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "star_table_2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Worse - Using whitespace as a delimiter\n",
    "\n",
    "##### `./Data/StarData_Ver3.dat`\n",
    "\n",
    "```\n",
    "StarID Parallax GMag \n",
    "A5853 768.07 8.98\n",
    "B4472 546.98 8.19\n",
    "C3864 415.18 11.03\n",
    "D7628 392.75 6.55\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "star_table_3 = QTable.read('./Data/StarData_Ver3.dat', \n",
    "                           format='ascii.csv',\n",
    "                           delimiter = \" \")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "star_table_3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "# Messy Data\n",
    "\n",
    "<img style=\"float: right;\" src=\"./images/MessyData.jpg\" width=\"230\"/>\n",
    "\n",
    "* In the \"real world\" all data is messy."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Let us read-in the file: `./Data/Messy.dat`\n",
    "\n",
    "```\n",
    "#######################################################\n",
    "#\n",
    "# Col 1 - Name\n",
    "# Col 2 - Distance\n",
    "#\n",
    "#######################################################\n",
    "\"A2344\",10\n",
    "\"\",23\n",
    ",\n",
    "# A random comment row just because\n",
    "\"E5333\",\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### This is not going to end well ... (errors galore!)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "messy_table = star_table = QTable.read('./Data/Messy.dat', \n",
    "                                       format='ascii.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Deal with the comment lines `#`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "messy_table = star_table = QTable.read('./Data/Messy.dat', \n",
    "                                       format='ascii.csv',\n",
    "                                       comment = '#')\n",
    "\n",
    "messy_table"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Not quite correct ..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Turn off the header\n",
    "\n",
    "- Since the first row is not the header we cannot use `ascii.csv`\n",
    "- Switch to `ascii.no_header`\n",
    "- add a delimiter "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "messy_table = star_table = QTable.read('./Data/Messy.dat', \n",
    "                                       format='ascii.no_header',\n",
    "                                       delimiter = ',',\n",
    "                                       comment = '#',)\n",
    "\n",
    "messy_table"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Add the column names"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_column_names = ['Name', 'Distance']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "messy_table = star_table = QTable.read('./Data/Messy.dat', \n",
    "                                       format='ascii.no_header',\n",
    "                                       delimiter = ',',\n",
    "                                       comment = '#',\n",
    "                                       names = my_column_names)\n",
    "\n",
    "messy_table"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Deal with the missing data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "messy_table['Name'].fill_value = 'XXXXX'\n",
    "messy_table['Distance'].fill_value = -999\n",
    "\n",
    "messy_table.filled()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "----\n",
    "\n",
    "# Fixed-Width Data Tables\n",
    "\n",
    "* These types of data tables are **VERY** common in astronomy\n",
    "* The columns have a fixed-widths\n",
    "* Whitespace is used to seperate columns **AND** used within columns\n"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "TPHE A      00 30 09\n",
    "PG0029+024  00 31 50\n",
    "92 309      00 53 14"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "  TPHE A      00 30 09\n",
    "  PG0029+024  00 31 50\n",
    "  92 309      00 53 14\n",
    "  ^           ^  ^  ^  \n",
    "  |----|----|----|----|\n",
    "  0    5    10   15   20\n",
    "  \n",
    "Notice that the four columns start at:\n",
    "\n",
    "Col 1 -> Index = 0\n",
    "Col 2 -> Index = 12\n",
    "Col 3 -> Index = 15\n",
    "Col 4 -> Index = 18"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_column_start = [0, 12, 15, 18]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_column_names = ['Star', 'RAh', 'RAm', 'RAs']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "standard_table = QTable.read('./Data/StdStars.dat', \n",
    "                            format='ascii.fixed_width_no_header',\n",
    "                            names = my_column_names,\n",
    "                            col_starts = my_column_start\n",
    "                            )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "standard_table"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "----\n",
    "\n",
    "# ⭐ Bonus Content ⭐\n",
    "\n",
    "# [CDS](https://cds.u-strasbg.fr/)\n",
    "\n",
    "### The Centre de Données astronomiques de Strasbourg (Strasbourg astronomical Data Center) is the main repository for published astronomical data as well as the host to SIMBAD the main astronomical object database.\n",
    "\n",
    "### All of the data published in the astronomical literature is availible in electronic form\n",
    "\n",
    "---\n",
    "\n",
    "![AJ](./images/StdStars.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Landolt Paper data page](https://vizier.cds.unistra.fr/viz-bin/VizieR?-source=II/183)"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "table2.dat"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "TPHE A      00 30 09 -46 31 22  14.651  0.793  0.380  0.435  0.405  0.841  29  12  0.0028 0.0046 0.0071 0.0019 0.0035 0.0032\n",
    "TPHE B      00 30 16 -46 27 55  12.334  0.405  0.156  0.262  0.271  0.535  29  17  0.0115 0.0026 0.0039 0.0020 0.0019 0.0035\n",
    "TPHE C      00 30 17 -46 32 34  14.376 -0.298 -1.217 -0.148 -0.211 -0.360  39  23  0.0022 0.0024 0.0043 0.0038 0.0133 0.0149"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "ReadMe"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "Byte-by-byte Description of file: table2.dat\n",
    "--------------------------------------------------------------------------------\n",
    "   Bytes Format Units   Label   Explanations\n",
    "--------------------------------------------------------------------------------\n",
    "   1- 11  A11   ---     Star   *Star Designation\n",
    "  13- 14  I2    h       RAh     Right Ascension J2000 (hours)\n",
    "  16- 17  I2    min     RAm     Right Ascension J2000 (minutes)\n",
    "  19- 20  I2    s       RAs     Right Ascension J2000 (seconds)\n",
    "      22  A1    ---     DE-     Declination J2000 (sign)\n",
    "  23- 24  I2    deg     DEd     Declination J2000 (degrees)\n",
    "  26- 27  I2    arcmin  DEm     Declination J2000 (minutes)\n",
    "  29- 30  I2    arcsec  DEs     Declination J2000 (seconds)\n",
    "  33- 38  F6.3  mag     Vmag    V magnitude\n",
    "  40- 45  F6.3  mag     B-V     B-V color\n",
    "  47- 52  F6.3  mag     U-B     U-B color\n",
    "  54- 59  F6.3  mag     V-R     V-R color\n",
    "  61- 66  F6.3  mag     R-I     R-I color\n",
    "  68- 73  F6.3  mag     V-I     V-I color\n",
    "  75- 77  I3    ---   o_Vmag    Number of observations\n",
    "  79- 81  I3    ---     Nd      Number of nights\n",
    "  84- 89  F6.4  mag   e_Vmag    Mean error of the Mean Vmag\n",
    "  91- 96  F6.4  mag   e_B-V     Mean error of the Mean (B-V)\n",
    "  98-103  F6.4  mag   e_U-B     Mean error of the Mean (U-B)\n",
    " 105-110  F6.4  mag   e_V-R     Mean error of the Mean (V-R)\n",
    " 112-117  F6.4  mag   e_R-I     Mean error of the Mean (R-I)\n",
    " 119-124  F6.4  mag   e_V-I     Mean error of the Mean (V-I)\n",
    "--------------------------------------------------------------------------------"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "standard_table = QTable.read(\"https://cdsarc.cds.unistra.fr/ftp/II/183A/table2.dat\",\n",
    "                             readme=\"https://cdsarc.cds.unistra.fr/ftp/II/183A/ReadMe\",\n",
    "                             format=\"ascii.cds\")  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "standard_table"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "----\n",
    "\n",
    "# Lots of Data\n",
    "\n",
    "<img style=\"float: right;\" src=\"./images/LotsData.jpg\" width=\"230\"/>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "standard_table.show_in_notebook()"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}