final-project-requirements.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "dominant-scale",
   "metadata": {},
   "source": [
    "# Project Requirements"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dramatic-preview",
   "metadata": {},
   "source": [
    "* We'll present projects as a group on Tuesday, April 13. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "large-norfolk",
   "metadata": {},
   "source": [
    "### Module 1 Requirements"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "alpine-grill",
   "metadata": {},
   "source": [
    "1. Build models and tables $3^{+}$\n",
    "2. Connect to API dataset $1^{+}$ \n",
    "    * With the adapter pattern (at least 1 API)\n",
    "3. Write Tests  $8^{+}$ \n",
    "4. Connect to Streamlit front end"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "friendly-tunnel",
   "metadata": {},
   "source": [
    "### Module 2 Requirements"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "moving-clarity",
   "metadata": {},
   "source": [
    "1. Connect a publicly accessible database\n",
    "\n",
    "* Create an RDS database, and make it publicly accessible\n",
    "* Run the migrations against the database with the `psql -f` flag\n",
    "* Connect the flask application to the RDS database\n",
    "* Move the DATABASE_URL to a `.env` file.  Make sure that any other variables (urls, api urls, keys, and secrets are also stored as environmental variables.\n",
    "* With code on local computer, can make a request to API, and fill database with some initial data\n",
    "\n",
    "2. Dockerize the application\n",
    "\n",
    "* Create the backend docker image\n",
    "* Create the docker image for the frontend\n",
    "* Check that can bootup frontend and backend docker containers\n",
    "* Confirm that making use of caching in Docker\n",
    "* Push to dockerhub\n",
    "\n",
    "3. Integrate kubernetes\n",
    "\n",
    "* Use kubernetes architecture of a backend service and frontend service\n",
    " * Use environmental variables and secrets in Kubernetes to set urls, and api information\n",
    "* Check that can both work properly, and can access from browser\n",
    "* And then work on ensuring that frontend connects to backend through services\n",
    "* Add an ingress service locally\n",
    "\n",
    "4. Deploy to AWS\n",
    "\n",
    "* Deploy to AWS by first setting up the cluster (can log into existing EC2 machine to do so)\n",
    "* Add the `deploy.yaml` file to add ingress nginx and load balancer\n",
    "* Clone the k8s files to the EC2 machine and create the corresponding resources\n",
    "* Bonus: Allow for access to the RDS database from security group of the two EC2 nodes, and then can make database private.m"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "vertical-violation",
   "metadata": {},
   "source": [
    "### Module 3 "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "allied-ebony",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "1. Create code for OLAP database in star schema to regularly migrate data from OLTP to OLAP\n",
    "    * First design the OLAP database\n",
    "    * Then create tables in redshift (consider distribution strategies of Even, All, Key)\n",
    "    * Then move to ETL in Airflow \n",
    "2. Automate calling of ETL with airflow (Bonus)\n",
    "    * To do so, will have to move the kickoff of ETL process to airflow container/dags \n",
    "    "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}