Python_ReadingData.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Reading Data\n",
    "\n",
    "<img src=\"https://uwashington-astro300.github.io/A300_images/DataReading.png\" width=\"120\"/>\n",
    "\n",
    "* Astropy has a large number of different ways to read data from external files. \n",
    "* Astropy supports almost any type of file you can think of, from simple text files to complex binary formats.\n",
    "* Most of our datafiles will be `csv` files (comma separated values)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from astropy.table import QTable"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Let us read-in the file `StarData.csv` via a URL\n",
    "- https://uwashington-astro300.github.io/A300_Data/StarData.csv\n",
    "\n",
    "```\n",
    "StarID,Parallax,GMag\n",
    "A5853,768.07,8.98\n",
    "B4472,546.98,8.19\n",
    "C3864,415.18,11.03\n",
    "D7628,392.75,6.55\n",
    "```\n",
    "\n",
    "### `ascii.csv` assumes the first row is a list of the column names"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "star_table = QTable.read('https://uwashington-astro300.github.io/A300_Data/StarData.csv', \n",
    "                         format='ascii.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "star_table"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "----\n",
    "\n",
    "# Reading (bad) Data\n",
    "\n",
    "<img src=\"https://uwashington-astro300.github.io/A300_images/Lore.jpg\" width=\"200\"/>\n",
    "\n",
    "## Different Delimiters\n",
    "\n",
    "Some people just want to watch the world burn, so they create datasets where the columns are separted by something other than a comma."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Bad - Using another delimiter like `:`\n",
    "\n",
    "##### `StarData_Ver2.dat`\n",
    "\n",
    "```\n",
    "StarID:Parallax:GMag\n",
    "A5853:768.07:8.98\n",
    "B4472:546.98:8.19\n",
    "C3864:415.18:11.03\n",
    "D7628:392.75:6.55\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "star_table_2 = QTable.read('https://uwashington-astro300.github.io/A300_Data/StarData_Ver2.dat', \n",
    "                           format='ascii.csv',\n",
    "                           delimiter = \":\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "star_table_2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Worse - Using whitespace as a delimiter\n",
    "\n",
    "##### `StarData_Ver3.dat`\n",
    "\n",
    "```\n",
    "StarID Parallax GMag \n",
    "A5853 768.07 8.98\n",
    "B4472 546.98 8.19\n",
    "C3864 415.18 11.03\n",
    "D7628 392.75 6.55\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "star_table_3 = QTable.read('https://uwashington-astro300.github.io/A300_Data/StarData_Ver3.dat', \n",
    "                           format='ascii.csv',\n",
    "                           delimiter = \" \")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "star_table_3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "# Messy Data\n",
    "\n",
    "<img src=\"https://uwashington-astro300.github.io/A300_images/MessyData.jpg\" width=\"230\"/>\n",
    "\n",
    "* In the \"real world\" all data is messy."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Let us read-in the file: `Messy.dat`\n",
    "\n",
    "```\n",
    "#######################################################\n",
    "#\n",
    "# Col 1 - Name\n",
    "# Col 2 - Distance\n",
    "#\n",
    "#######################################################\n",
    "\"A2344\",10\n",
    "\"\",23\n",
    ",\n",
    "# A random comment row just because\n",
    "\"E5333\",\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### This is not going to end well ... (errors galore!)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "messy_table = star_table = QTable.read('https://uwashington-astro300.github.io/A300_Data/Messy.dat', \n",
    "                                       format='ascii.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Deal with the comment lines `#`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "messy_table = star_table = QTable.read('https://uwashington-astro300.github.io/A300_Data/Messy.dat', \n",
    "                                       format='ascii.csv',\n",
    "                                       comment = '#')\n",
    "\n",
    "messy_table"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Not quite correct ..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Turn off the header\n",
    "\n",
    "- Since the first row is not the header we cannot use `ascii.csv`\n",
    "- Switch to `ascii.no_header`\n",
    "- add a delimiter "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "messy_table = QTable.read('https://uwashington-astro300.github.io/A300_Data/Messy.dat', \n",
    "                          format='ascii.no_header',\n",
    "                          delimiter = ',',\n",
    "                          comment = '#',)\n",
    "\n",
    "messy_table"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Add the column names"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_column_names = ['Name', 'Distance']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "messy_table = QTable.read('https://uwashington-astro300.github.io/A300_Data/Messy.dat', \n",
    "                          format='ascii.no_header',\n",
    "                          delimiter = ',',\n",
    "                          comment = '#',\n",
    "                          names = my_column_names)\n",
    "\n",
    "messy_table"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Deal with the missing data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "messy_table['Name'].fill_value = 'XXXXX'\n",
    "messy_table['Distance'].fill_value = -999\n",
    "\n",
    "messy_table.filled()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "----\n",
    "\n",
    "# Fixed-Width Data Tables\n",
    "\n",
    "* These types of data tables are **VERY** common in astronomy\n",
    "* The columns have a fixed-widths\n",
    "* Whitespace is used to seperate columns **AND** used within columns\n",
    "\n",
    "`StdStars.dat`"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "TPHE A      00 30 09\n",
    "PG0029+024  00 31 50\n",
    "92 309      00 53 14"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "  TPHE A      00 30 09\n",
    "  PG0029+024  00 31 50\n",
    "  92 309      00 53 14\n",
    "  ^           ^  ^  ^  \n",
    "  |----|----|----|----|\n",
    "  0    5    10   15   20\n",
    "  \n",
    "Notice that the four columns start at:\n",
    "\n",
    "Col 1 -> Index = 0\n",
    "Col 2 -> Index = 12\n",
    "Col 3 -> Index = 15\n",
    "Col 4 -> Index = 18"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_column_start = [0, 12, 15, 18]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_column_names = ['Star', 'RAh', 'RAm', 'RAs']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "standard_table = QTable.read('https://uwashington-astro300.github.io/A300_Data/StdStars.dat', \n",
    "                            format='ascii.fixed_width_no_header',\n",
    "                            names = my_column_names,\n",
    "                            col_starts = my_column_start\n",
    "                            )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "standard_table"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "----\n",
    "\n",
    "# Lots of Data\n",
    "\n",
    "<img  src=\"https://uwashington-astro300.github.io/A300_images/LotsData.jpg\" width=\"230\"/>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "temp_table = QTable.read('https://uwashington-astro300.github.io/A300_Data/SeattleTemp_2022.csv', \n",
    "                         format='ascii.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "temp_table"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "temp_table.show_in_notebook()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}