diff --git a/pyspark/Colab and PySpark.ipynb b/pyspark/Colab and PySpark.ipynb index 7fc0558..78f62bd 100644 --- a/pyspark/Colab and PySpark.ipynb +++ b/pyspark/Colab and PySpark.ipynb @@ -1,3073 +1,4158 @@ { - "cells": [ - { - "cell_type": "markdown", - "metadata": { - "colab_type": "text", - "id": "qJoeN3e8_Gzk" - }, - "source": [ - "

Introduction to Google Colab and PySpark

" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Table Of Contents:\n", - "
    \n", - "
  1. Objective
    2. \n", - "
  2. Prerequisite
    3. \n", - "
  3. Notes from the author
    4. \n", - "
  4. Big data, PySpark and Colaboratory
    5. \n", - "
      \n", - "
    1. Big data
      2. \n", - "
    2. PySpark
      3. \n", - "
    3. Colaboratory
      4. \n", - "
    \n", - "
  5. Jupyter notebook basics
    6. \n", - "
      \n", - "
    1. Code cells
      2. \n", - "
    2. Text cells
      3. \n", - "
    3. Access to the shell
      4. \n", - "
    4. Install Spark
      5. \n", - "
    \n", - "
  6. Loading Dataset
    7. \n", - "
  7. Working with the DataFrame API
    8. \n", - "
      \n", - "
    1. Viewing Dataframe
      2. \n", - "
    2. Schema of a DataFrame
      3. \n", - "
    3. Working with columns
      4. \n", - "
    4. Working with Rows
      5. \n", - "
    \n", - "
  8. Hands-on Questions
    9. \n", - "
  9. Functions
    10. \n", - "
      \n", - "
    1. String Functions
      2. \n", - "
    2. Numeric functions
      3. \n", - "
    3. Date
      4. \n", - "
    \n", - "
  10. Working with Dates
    11. \n", - "
  11. Working with joins
    12. \n", - "
  12. Hands-on again!
    13. \n", - "
  13. RDD
    14. \n", - "
  14. User-Defined Functions (UDF)
    15. \n", - "
  15. Common Questions
    16. \n", - "
" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "\n", - "## Objective\n", - "The objective of this notebook is to:\n", - ">
  • Give a proper understanding about the different PySpark functions available.
    • \n", - ">
  • A short introduction to Google Colab, as that is the platform on which this notebook is written on.
    • \n", - "\n", - "Once you complete this notebook, you should be able to write pyspark programs in an efficent way. The ideal way to use this is by going through the examples given and then trying them on Colab. At the end there are a few hands on questions which you can use to evaluate yourself." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "\n", - "## Prerequisite\n", - ">
  • Although some theory about pyspark and big data will be given in this notebook, I recommend everyone to read more about it and have a deeper understanding on how the functions get executed and the relevance of big data in the current scenario.\n", - ">
  • A good understanding on python will be an added bonus." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "\n", - "### Notes from the author\n", - "\n", - "This tutorial was made using Google Colab so the code you see here is meant to run on a colab notebook.
    \n", - "It goes through basic [PySpark Functions](https://spark.apache.org/docs/latest/api/python/index.html) and a short introduction on how to use [Colab](https://colab.research.google.com/notebooks/basic_features_overview.ipynb).
    \n", - "If you want to view my colab notebook for this particular tutorial, you can view it [here](https://colab.research.google.com/drive/1G894WS7ltIUTusWWmsCnF_zQhQqZCDOc). The viewing experience and readability is much better there.
    \n", - "If you want to try out things with this notebook as a base, feel free to download it from my repo [here](https://github.com/jacobceles/knowledge-repo/blob/master/pyspark/Colab%20and%20PySpark.ipynb) and then use it with jupyter notebook." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "\n", - "## Big data, PySpark and Colaboratory" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "\n", - "### Big data" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Big data usually means data of such huge volume that normal data storage solutions cannot efficently store and process it. In this era, data is being generated at an absurd rate. Data is collected for each movement a person makes. The bulk of big data comes from three primary sources: \n", - "
      \n", - "
    1. Social data
      2. \n", - "
    2. Machine data
      3. \n", - "
    3. Transactional data
      4. \n", - "
    \n", - "\n", - "Some common examples for the sources of such data include internet searches, facebook posts, doorbell cams, smartwatches, online shopping history etc. Every action creates data, it is just a matter of of there is a way to collect them or not. But what's interesting is that out of all this data collected, not even 5% of it is being used fully. There is a huge demand for big data professionals in the industry. Even though the number of graduates with a specialization in big data are rising, the problem is that they don't have the practical knowledge about big data scenarios, which leads to bad architecutres and inefficent methods of processing data.\n", - "\n", - ">If you are interested to know more about the landscape and technologies involved, here is [an article](https://hostingtribunal.com/blog/big-data-stats/) which I found really interesting!" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "\n", - "### PySpark" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "If you are working in the field of big data, you must have definelty heard of spark. If you look at the [Apache Spark](https://spark.apache.org/) website, you will see that it is said to be a `Lightning-fast unified analytics engine`. PySPark is a flavour of Spark used for processing and analysing massive volumes of data. If you are familiar with python and have tried it for huge datasets, you should know that the execution time can get ridiculous. Enter PySpark!\n", - "\n", - "Imagine your data resides in a distributed manner at different places. If you try brining your data to one point and executing your code there, not only would that be inefficent, but also cause memory issues. Now let's say your code goes to the data rather than the data coming to where your code. This will help avoid unneccesary data movement which will thereby decrease the running time. \n", - "\n", - "PySpark is the Python API of Spark; which means it can do almost all the things python can. Machine learning(ML) pipelines, exploratory data analysis (at scale), ETLs for data platform, and much more! And all of them in a distributed manner. One of the best parts of pyspark is that if you are already familiar with python, it's really easy to learn.\n", - "\n", - "Apart from PySpark, there is another language called Scala used for big data processing. Scala is frequently over 10 times faster than Python is native for Hadoop as its based on JVM. But PySpark is getting adopted at a fast rate because of the ease of use, easier learning curve and ML capabilities.\n", - "\n", - "I will briefly explain how a PySpark job works, but I strongly recommend you read more about the [architecture](https://data-flair.training/blogs/how-apache-spark-works/) and how everything works. Now, before I get into it, let me talk about some basic jargons first:\n", - "\n", - "Cluster is a set of loosely or tightly connected computers that work together so that they can be viewed as a single system.\n", - "\n", - "Hadoop is an open source, scalable, and fault tolerant framework written in Java. It efficiently processes large volumes of data on a cluster of commodity hardware. Hadoop is not only a storage system but is a platform for large data storage as well as processing.\n", - "\n", - "HDFS (Hadoop distributed file system). It is one of the world's most reliable storage system. HDFS is a Filesystem of Hadoop designed for storing very large files running on a cluster of commodity hardware.\n", - "\n", - "MapReduce is a data Processing framework, which has 2 phases - Mapper and Reducer. The map procedure performs filtering and sorting, and the reduce method performs a summary operation. It usually runs on a hadoop cluster.\n", - "\n", - "Transformation refers to the operations applied on a dataset to create a new dataset. Filter, groupBy and map are the examples of transformations.\n", - "\n", - "Actions Actions refer to an operation which instructs Spark to perform computation and send the result back to driver. This is an example of action.\n", - "\n", - "Alright! Now that that's out of the way, let me explain how a spark job runs. In simple terma, each time you submit a pyspark job, the code gets internally converted into a MapReduce program and gets executed in the Java virtual machine. Now one of the thoughts that might be popping in your mind will probably be:
    `So the code gets converted into a MapReduce program. Wouldn't that mean MapReduce is faster than pySpark?`
    Well, the answer is a big NO. This is what makes spark jobs special. Spark is capable of handling a massive amount of data at a time, in it's distributed environment. It does this through in-memory processing, which is what makes it almost 100 times faster than Hadoop. Another factor which amkes it fast is Lazy Evaluation. Spark delays its evaluation as much as it can. Each time you submit a job, spark creates an action plan for how to execute the code, and then does nothing. Finally, when you ask for the result(i.e, calls an action), it executes the plan, which is basically all the transofrmations you have mentioned in your code. That's basically the gist of it. \n", - "\n", - "Now lastly, I want to talk about on more thing. Spark mainly consists of 4 modules:\n", - "\n", - "
      \n", - "
    1. Spark SQL - helps to write spark programs using SQL like queries.
      2. \n", - "
    2. Spark Streaming - is an extension of the core Spark API that enables scalable, high-throughput, fault-tolerant stream processing of live data streams. used heavily in processing of social media data.
      3. \n", - "
    3. Spark MLLib - is the machine learning component of SPark. It helps train ML models on massive datasets with very high efficeny.
      4. \n", - "
    4. Spark GraphX - is the visualization component of Spark. It enables users to view data both as graphs and as collections without data movement or duplication.
      5. \n", - "
    \n", - "\n", - "Hopefully this image gives a better idea of what I am talking about:\n", - "\"Spark\n", - "
    Source: Datanami
    \n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "\n", - "### Colaboratory" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "In the words of Google:
    \n", - "`Colaboratory, or “Colab” for short, is a product from Google Research. Colab allows anybody to write and execute arbitrary python code through the browser, and is especially well suited to machine learning, data analysis and education. More technically, Colab is a hosted Jupyter notebook service that requires no setup to use, while providing free access to computing resources including GPUs.`\n", - "\n", - "The reason why I used colab is because of its shareability and free GPU. Yeah you read that right. A FREE GPU! Additionally, it helps use different Google services conveniently. It saves to Google Drive and all the services are very closely related. I recommend you go through the offical [overview documentation](https://colab.research.google.com/notebooks/basic_features_overview.ipynb) if you want to know more about it.\n", - "If you have more questions about colab, please [refer this link](https://research.google.com/colaboratory/faq.html)\n", - "\n", - ">While using a colab notebook, you will need an active internet connection to keep a session alive. If you lose the connection you will have to download the datasets again." - ] - }, - { - "cell_type": "markdown", - "metadata": { - "colab_type": "text", - "id": "_N5-lspH_N8B" - }, - "source": [ - "\n", - "## Jupyter notebook basics" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "colab_type": "text", - "id": "6Ul54hAYyHyd" - }, - "source": [ - "\n", - "### Code cells" - ] - }, - { - "cell_type": "code", - "execution_count": 1, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 34 - }, - "colab_type": "code", - "id": "j38beRUTCI5c", - "outputId": "0aba3c8e-8741-411e-c771-9f8440f7b77f" - }, - "outputs": [ - { - "data": { - "text/plain": [ - "6" - ] - }, - "execution_count": 1, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "2*3" - ] - }, - { - "cell_type": "code", - "execution_count": 2, - "metadata": { - "colab": {}, - "colab_type": "code", - "id": "_Jewe_e9CIYa" - }, - "outputs": [], - "source": [ - "import pandas as pd" - ] - }, - { - "cell_type": "code", - "execution_count": 3, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 34 - }, - "colab_type": "code", - "id": "g8Y7w6_CCIIT", - "outputId": "5b147862-bdac-4388-b08f-fdf353570d09" - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Hello!\n" - ] - } - ], - "source": [ - "print(\"Hello!\")" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "colab_type": "text", - "id": "VOqLNkRKyUIS" - }, - "source": [ - "\n", - "### Text cells" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Hello world!" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "colab_type": "text", - "id": "X6zdrH15_CCW" - }, - "source": [ - "\n", - "## Access to the shell" - ] - }, - { - "cell_type": "code", - "execution_count": 4, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 85 - }, - "colab_type": "code", - "id": "zdO9sjSdEVnr", - "outputId": "84681aa2-a37c-447c-a082-a55071e80796" - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "\u001b[34mBackup DAGs\u001b[m\u001b[m/\r\n", - "Colab_and_PySpark.ipynb\r\n", - "\u001b[34mReference\u001b[m\u001b[m/\r\n", - "~$ctory Audit_AS-IS Process_03142019.docx\r\n" - ] - } - ], - "source": [ - "ls" - ] - }, - { - "cell_type": "code", - "execution_count": 5, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 34 - }, - "colab_type": "code", - "id": "QF9e3lDDEX3I", - "outputId": "63b0236d-8157-4486-a190-26f15a570726" - }, - "outputs": [ - { - "data": { - "text/plain": [ - "u'/Users/jcele1/Downloads'" - ] - }, - "execution_count": 5, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "pwd" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "colab_type": "text", - "id": "Dd6t0uFzuR4X" - }, - "source": [ - "\n", - "## Install Spark" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 493 - }, - "colab_type": "code", - "id": "tt7ZS1_wGgjn", - "outputId": "aa62b40f-f9bd-4b58-f0ab-7487914ac06e" - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "/bin/sh: apt-get: command not found\n", - "/bin/sh: apt-get: command not found\n" - ] - } - ], - "source": [ - "!apt-get update\n", - "!apt-get install openjdk-8-jdk-headless -qq > /dev/null\n", - "!wget -q http://archive.apache.org/dist/spark/spark-2.4.4/spark-2.4.4-bin-hadoop2.7.tgz\n", - "!tar xf spark-2.4.4-bin-hadoop2.7.tgz\n", - "!pip install -q findspark" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "colab": {}, - "colab_type": "code", - "id": "sdOOq4twHN1K" - }, - "outputs": [], - "source": [ - "import os\n", - "os.environ[\"JAVA_HOME\"] = \"/usr/lib/jvm/java-8-openjdk-amd64\"\n", - "os.environ[\"SPARK_HOME\"] = \"/content/spark-2.4.4-bin-hadoop2.7\"" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 34 - }, - "colab_type": "code", - "id": "3ACYMwhgHTYz", - "outputId": "e7e5ce2b-cec9-4537-c327-007ecd69b9d0" - }, - "outputs": [], - "source": [ - "!ls" - ] - }, - { - "cell_type": "code", - "execution_count": 0, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 193 - }, - "colab_type": "code", - "id": "wjfF7LLgHZe3", - "outputId": "043fdfeb-38a1-400f-d8a6-1e371962514a" - }, - "outputs": [ - { - "data": { - "text/html": [ - "\n", - "
    \n", - "

    SparkContext

    \n", - "\n", - "

    Spark UI

    \n", - "\n", - "
    \n", - "
    Version
    \n", - "
    v2.4.4
    \n", - "
    Master
    \n", - "
    local[*]
    \n", - "
    AppName
    \n", - "
    pyspark-shell
    \n", - "
    \n", - "
    \n", - " " - ], - "text/plain": [ - "" - ] - }, - "execution_count": 4, - "metadata": { - "tags": [] - }, - "output_type": "execute_result" - } - ], - "source": [ - "import findspark\n", - "findspark.init()\n", - "from pyspark import SparkContext\n", - "\n", - "sc = SparkContext.getOrCreate()\n", - "sc" - ] - }, - { - "cell_type": "code", - "execution_count": 0, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 216 - }, - "colab_type": "code", - "id": "Gs7fzvxcHfvw", - "outputId": "757dc23b-d3ba-4b67-ead7-e654ef674db3" - }, - "outputs": [ - { - "data": { - "text/html": [ - "\n", - "
    \n", - "

    SparkSession - in-memory

    \n", - " \n", - "
    \n", - "

    SparkContext

    \n", - "\n", - "

    Spark UI

    \n", - "\n", - "
    \n", - "
    Version
    \n", - "
    v2.4.4
    \n", - "
    Master
    \n", - "
    local[*]
    \n", - "
    AppName
    \n", - "
    pyspark-shell
    \n", - "
    \n", - "
    \n", - " \n", - "
    \n", - " " - ], - "text/plain": [ - "" - ] - }, - "execution_count": 6, - "metadata": { - "tags": [] - }, - "output_type": "execute_result" - } - ], - "source": [ - "import pyspark\n", - "from pyspark.sql import SparkSession\n", - "spark = SparkSession.builder.getOrCreate() \n", - "spark" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "colab_type": "text", - "id": "hmIqq6xPK7m7" - }, - "source": [ - "\n", - "# Loading Dataset" - ] - }, - { - "cell_type": "code", - "execution_count": 0, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 204 - }, - "colab_type": "code", - "id": "hQ3zmGACLKlN", - "outputId": "0948af88-5021-45b3-a966-3081c6c3f022" - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "--2020-01-22 04:20:04-- https://data.cityofchicago.org/api/views/w98m-zvie/rows.csv?accessType=DOWNLOAD\n", - "Resolving data.cityofchicago.org (data.cityofchicago.org)... 52.206.140.199, 52.206.68.26, 52.206.140.205\n", - "Connecting to data.cityofchicago.org (data.cityofchicago.org)|52.206.140.199|:443... connected.\n", - "HTTP request sent, awaiting response... 200 OK\n", - "Length: unspecified [text/csv]\n", - "Saving to: ‘rows.csv?accessType=DOWNLOAD’\n", - "\n", - "rows.csv?accessType [ <=> ] 58.56M 3.32MB/s in 18s \n", - "\n", - "2020-01-22 04:20:23 (3.29 MB/s) - ‘rows.csv?accessType=DOWNLOAD’ saved [61404826]\n", - "\n" - ] - } - ], - "source": [ - "# Downloading and preprocessing Chicago's Reported Crime Data\n", - "!wget https://data.cityofchicago.org/api/views/w98m-zvie/rows.csv?accessType=DOWNLOAD" - ] - }, - { - "cell_type": "code", - "execution_count": 0, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 51 - }, - "colab_type": "code", - "id": "Wpq2jYvIMOJy", - "outputId": "16ee6f3a-5455-483c-dc06-54c19c55d59a" - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "'rows.csv?accessType=DOWNLOAD'\t spark-2.4.4-bin-hadoop2.7\n", - " sample_data\t\t\t spark-2.4.4-bin-hadoop2.7.tgz\n" - ] - } - ], - "source": [ - "!ls" - ] - }, - { - "cell_type": "code", - "execution_count": 0, - "metadata": { - "colab": {}, - "colab_type": "code", - "id": "uCJX5cwdMS9q" - }, - "outputs": [], - "source": [ - "#Renaming the downloaded file\n", - "!mv rows.csv?accessType=DOWNLOAD reported-crimes.csv" - ] - }, - { - "cell_type": "code", - "execution_count": 0, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 204 - }, - "colab_type": "code", - "id": "hz6ALr5mMqZt", - "outputId": "71469489-37e7-4ef6-a483-2332ef0d3cf8" - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "+--------+-----------+--------------------+--------------------+----+--------------------+--------------------+--------------------+------+--------+----+--------+----+--------------+--------+------------+------------+----+--------------------+--------+---------+--------+\n", - "| ID|Case Number| Date| Block|IUCR| Primary Type| Description|Location Description|Arrest|Domestic|Beat|District|Ward|Community Area|FBI Code|X Coordinate|Y Coordinate|Year| Updated On|Latitude|Longitude|Location|\n", - "+--------+-----------+--------------------+--------------------+----+--------------------+--------------------+--------------------+------+--------+----+--------+----+--------------+--------+------------+------------+----+--------------------+--------+---------+--------+\n", - "|11955940| JD121140|07/31/2019 09:00:...| 026XX S HALSTED ST|1154| DECEPTIVE PRACTICE|FINANCIAL IDENTIT...| SMALL RETAIL STORE| false| false|0913| 009| 11| 60| 11| null| null|2019|01/20/2020 03:52:...| null| null| null|\n", - "|11956035| JD121288|09/10/2019 12:01:...|070XX S STONY ISL...|1582|OFFENSE INVOLVING...| CHILD PORNOGRAPHY| RESIDENCE| false| false|0332| 003| 5| 43| 17| null| null|2019|01/20/2020 03:52:...| null| null| null|\n", - "|11956045| JD121237|12/15/2019 12:00:...| 069XX S ADA ST|0610| BURGLARY| FORCIBLE ENTRY| RESIDENCE| false| false|0734| 007| 6| 67| 05| null| null|2019|01/20/2020 03:52:...| null| null| null|\n", - "|11956146| JD121344|12/31/2019 12:00:...| 016XX S THROOP ST|0610| BURGLARY| FORCIBLE ENTRY| RESIDENCE| false| false|1233| 012| 25| 31| 05| null| null|2019|01/20/2020 03:52:...| null| null| null|\n", - "|11956304| JD121233|11/27/2019 10:35:...| 002XX W GOETHE ST|0810| THEFT| OVER $500| RESIDENCE| false| false|1821| 018| 2| 8| 06| null| null|2019|01/20/2020 03:52:...| null| null| null|\n", - "+--------+-----------+--------------------+--------------------+----+--------------------+--------------------+--------------------+------+--------+----+--------+----+--------------+--------+------------+------------+----+--------------------+--------+---------+--------+\n", - "only showing top 5 rows\n", - "\n" - ] - } - ], - "source": [ - "df = spark.read.csv('reported-crimes.csv',header=True)\n", - "df.show(5)" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "colab_type": "text", - "id": "HgwoX-pfNqQI" - }, - "source": [ - "\n", - "# Working with the Dataframe API" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "colab_type": "text", - "id": "_QwZtWxZRCBn" - }, - "source": [ - "\n", - "## Viewing Dataframe" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "colab_type": "text", - "id": "50LZ3S8_PMg_" - }, - "source": [ - "In Spark, you have a couple of options to view the DataFrame(DF).\n", - "\n", - "\n", - "1. take(3) will return a list of three row objects. \n", - "2. df.collect() will get all of the data from the entire DataFrame . Be careful when using it, because if you have a large data set when you run collect, you can easily crash the driver node. \n", - "3. If you want Spark to print out your DataFrame in a nice format, then try df.show() with the number of rows as paramter. \n", - "\n", - "N.B: The limit function **returns a new DataFrame** by taking the first n rows." - ] - }, - { - "cell_type": "markdown", - "metadata": { - "colab_type": "text", - "id": "eFoagdqARKb8" - }, - "source": [ - "\n", - "## Schema of a DataFrame" - ] - }, - { - "cell_type": "code", - "execution_count": 0, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 391 - }, - "colab_type": "code", - "id": "w6qwTjGsNxrw", - "outputId": "fa18fcce-a7d0-41f3-934e-520d51eaadd1" - }, - "outputs": [ - { - "data": { - "text/plain": [ - "[('ID', 'string'),\n", - " ('Case Number', 'string'),\n", - " ('Date', 'string'),\n", - " ('Block', 'string'),\n", - " ('IUCR', 'string'),\n", - " ('Primary Type', 'string'),\n", - " ('Description', 'string'),\n", - " ('Location Description', 'string'),\n", - " ('Arrest', 'string'),\n", - " ('Domestic', 'string'),\n", - " ('Beat', 'string'),\n", - " ('District', 'string'),\n", - " ('Ward', 'string'),\n", - " ('Community Area', 'string'),\n", - " ('FBI Code', 'string'),\n", - " ('X Coordinate', 'string'),\n", - " ('Y Coordinate', 'string'),\n", - " ('Year', 'string'),\n", - " ('Updated On', 'string'),\n", - " ('Latitude', 'string'),\n", - " ('Longitude', 'string'),\n", - " ('Location', 'string')]" - ] - }, - "execution_count": 11, - "metadata": { - "tags": [] - }, - "output_type": "execute_result" - } - ], - "source": [ - "df.dtypes" - ] - }, - { - "cell_type": "code", - "execution_count": 0, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 425 - }, - "colab_type": "code", - "id": "CCGTFlCWRPw4", - "outputId": "4d85a813-406b-43bf-fd77-f6df935f3b42" - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "root\n", - " |-- ID: string (nullable = true)\n", - " |-- Case Number: string (nullable = true)\n", - " |-- Date: string (nullable = true)\n", - " |-- Block: string (nullable = true)\n", - " |-- IUCR: string (nullable = true)\n", - " |-- Primary Type: string (nullable = true)\n", - " |-- Description: string (nullable = true)\n", - " |-- Location Description: string (nullable = true)\n", - " |-- Arrest: string (nullable = true)\n", - " |-- Domestic: string (nullable = true)\n", - " |-- Beat: string (nullable = true)\n", - " |-- District: string (nullable = true)\n", - " |-- Ward: string (nullable = true)\n", - " |-- Community Area: string (nullable = true)\n", - " |-- FBI Code: string (nullable = true)\n", - " |-- X Coordinate: string (nullable = true)\n", - " |-- Y Coordinate: string (nullable = true)\n", - " |-- Year: string (nullable = true)\n", - " |-- Updated On: string (nullable = true)\n", - " |-- Latitude: string (nullable = true)\n", - " |-- Longitude: string (nullable = true)\n", - " |-- Location: string (nullable = true)\n", - "\n" - ] - } - ], - "source": [ - "df.printSchema()" - ] - }, - { - "cell_type": "code", - "execution_count": 0, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 391 - }, - "colab_type": "code", - "id": "xpsaQ4JMRUiS", - "outputId": "f4a5a4c7-b69a-49e3-9609-b8c5720ac2e2" - }, - "outputs": [ - { - "data": { - "text/plain": [ - "['ID',\n", - " 'Case Number',\n", - " 'Date',\n", - " 'Block',\n", - " 'IUCR',\n", - " 'Primary Type',\n", - " 'Description',\n", - " 'Location Description',\n", - " 'Arrest',\n", - " 'Domestic',\n", - " 'Beat',\n", - " 'District',\n", - " 'Ward',\n", - " 'Community Area',\n", - " 'FBI Code',\n", - " 'X Coordinate',\n", - " 'Y Coordinate',\n", - " 'Year',\n", - " 'Updated On',\n", - " 'Latitude',\n", - " 'Longitude',\n", - " 'Location']" - ] - }, - "execution_count": 13, - "metadata": { - "tags": [] - }, - "output_type": "execute_result" - } - ], - "source": [ - "# Defining a schema\n", - "from pyspark.sql.types import StructType, StructField, StringType, TimestampType, BooleanType, DoubleType, IntegerType\n", - "df.columns" - ] - }, - { - "cell_type": "code", - "execution_count": 0, - "metadata": { - "colab": {}, - "colab_type": "code", - "id": "ik62VX34SlFh" - }, - "outputs": [], - "source": [ - "labels = [\n", - " ('ID',StringType()),\n", - " ('Case Number',StringType()),\n", - " ('Date',TimestampType()),\n", - " ('Block',StringType()),\n", - " ('IUCR',StringType()),\n", - " ('Primary Type',StringType()),\n", - " ('Description',StringType()),\n", - " ('Location Description',StringType()),\n", - " ('Arrest',StringType()),\n", - " ('Domestic',BooleanType()),\n", - " ('Beat',StringType()),\n", - " ('District',StringType()),\n", - " ('Ward',StringType()),\n", - " ('Community Area',StringType()),\n", - " ('FBI Code',StringType()),\n", - " ('X Coordinate',StringType()),\n", - " ('Y Coordinate',StringType()),\n", - " ('Year',IntegerType()),\n", - " ('Updated On',StringType()),\n", - " ('Latitude',DoubleType()),\n", - " ('Longitude',DoubleType()),\n", - " ('Location',StringType()),\n", - " ('Historical Wards 2003-2015',StringType()),\n", - " ('Zip Codes',StringType()),\n", - " ('Community Areas',StringType()),\n", - " ('Census Tracts',StringType()),\n", - " ('Wards',StringType()),\n", - " ('Boundaries - ZIP Codes',StringType()),\n", - " ('Police Districts',StringType()),\n", - " ('Police Beats',StringType())\n", - "]" - ] - }, - { - "cell_type": "code", - "execution_count": 0, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 54 - }, - "colab_type": "code", - "id": "T-Fp5y_oU9SF", - "outputId": "b60ae500-9efa-463a-965f-b1d37bee6b97" - }, - "outputs": [ - { - "data": { - "text/plain": [ - "StructType(List(StructField(ID,StringType,true),StructField(Case Number,StringType,true),StructField(Date,TimestampType,true),StructField(Block,StringType,true),StructField(IUCR,StringType,true),StructField(Primary Type,StringType,true),StructField(Description,StringType,true),StructField(Location Description,StringType,true),StructField(Arrest,StringType,true),StructField(Domestic,BooleanType,true),StructField(Beat,StringType,true),StructField(District,StringType,true),StructField(Ward,StringType,true),StructField(Community Area,StringType,true),StructField(FBI Code,StringType,true),StructField(X Coordinate,StringType,true),StructField(Y Coordinate,StringType,true),StructField(Year,IntegerType,true),StructField(Updated On,StringType,true),StructField(Latitude,DoubleType,true),StructField(Longitude,DoubleType,true),StructField(Location,StringType,true),StructField(Historical Wards 2003-2015,StringType,true),StructField(Zip Codes,StringType,true),StructField(Community Areas,StringType,true),StructField(Census Tracts,StringType,true),StructField(Wards,StringType,true),StructField(Boundaries - ZIP Codes,StringType,true),StructField(Police Districts,StringType,true),StructField(Police Beats,StringType,true)))" - ] - }, - "execution_count": 15, - "metadata": { - "tags": [] - }, - "output_type": "execute_result" - } - ], - "source": [ - "schema = StructType([StructField (x[0], x[1], True) for x in labels])\n", - "schema" - ] - }, - { - "cell_type": "code", - "execution_count": 0, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 561 - }, - "colab_type": "code", - "id": "sgC7gtL5VTls", - "outputId": "55c6dcad-4127-4a49-b5e8-df884baa1023" - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "root\n", - " |-- ID: string (nullable = true)\n", - " |-- Case Number: string (nullable = true)\n", - " |-- Date: timestamp (nullable = true)\n", - " |-- Block: string (nullable = true)\n", - " |-- IUCR: string (nullable = true)\n", - " |-- Primary Type: string (nullable = true)\n", - " |-- Description: string (nullable = true)\n", - " |-- Location Description: string (nullable = true)\n", - " |-- Arrest: string (nullable = true)\n", - " |-- Domestic: boolean (nullable = true)\n", - " |-- Beat: string (nullable = true)\n", - " |-- District: string (nullable = true)\n", - " |-- Ward: string (nullable = true)\n", - " |-- Community Area: string (nullable = true)\n", - " |-- FBI Code: string (nullable = true)\n", - " |-- X Coordinate: string (nullable = true)\n", - " |-- Y Coordinate: string (nullable = true)\n", - " |-- Year: integer (nullable = true)\n", - " |-- Updated On: string (nullable = true)\n", - " |-- Latitude: double (nullable = true)\n", - " |-- Longitude: double (nullable = true)\n", - " |-- Location: string (nullable = true)\n", - " |-- Historical Wards 2003-2015: string (nullable = true)\n", - " |-- Zip Codes: string (nullable = true)\n", - " |-- Community Areas: string (nullable = true)\n", - " |-- Census Tracts: string (nullable = true)\n", - " |-- Wards: string (nullable = true)\n", - " |-- Boundaries - ZIP Codes: string (nullable = true)\n", - " |-- Police Districts: string (nullable = true)\n", - " |-- Police Beats: string (nullable = true)\n", - "\n" - ] - } - ], - "source": [ - "df = spark.read.csv('reported-crimes.csv',schema=schema)\n", - "df.printSchema()\n", - "# The schema comes as we gave!" - ] - }, - { - "cell_type": "code", - "execution_count": 0, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 479 - }, - "colab_type": "code", - "id": "Dn2EAhesVmx0", - "outputId": "285aaf1c-0946-4c74-d5e1-0e51a701f19e" - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "+----+-----------+----+-----+----+------------+-----------+--------------------+------+--------+----+--------+----+--------------+--------+------------+------------+----+----------+--------+---------+--------+--------------------------+---------+---------------+-------------+-----+----------------------+----------------+------------+\n", - "| ID|Case Number|Date|Block|IUCR|Primary Type|Description|Location Description|Arrest|Domestic|Beat|District|Ward|Community Area|FBI Code|X Coordinate|Y Coordinate|Year|Updated On|Latitude|Longitude|Location|Historical Wards 2003-2015|Zip Codes|Community Areas|Census Tracts|Wards|Boundaries - ZIP Codes|Police Districts|Police Beats|\n", - "+----+-----------+----+-----+----+------------+-----------+--------------------+------+--------+----+--------+----+--------------+--------+------------+------------+----+----------+--------+---------+--------+--------------------------+---------+---------------+-------------+-----+----------------------+----------------+------------+\n", - "|null| null|null| null|null| null| null| null| null| null|null| null|null| null| null| null| null|null| null| null| null| null| null| null| null| null| null| null| null| null|\n", - "|null| null|null| null|null| null| null| null| null| null|null| null|null| null| null| null| null|null| null| null| null| null| null| null| null| null| null| null| null| null|\n", - "|null| null|null| null|null| null| null| null| null| null|null| null|null| null| null| null| null|null| null| null| null| null| null| null| null| null| null| null| null| null|\n", - "|null| null|null| null|null| null| null| null| null| null|null| null|null| null| null| null| null|null| null| null| null| null| null| null| null| null| null| null| null| null|\n", - "|null| null|null| null|null| null| null| null| null| null|null| null|null| null| null| null| null|null| null| null| null| null| null| null| null| null| null| null| null| null|\n", - "|null| null|null| null|null| null| null| null| null| null|null| null|null| null| null| null| null|null| null| null| null| null| null| null| null| null| null| null| null| null|\n", - "|null| null|null| null|null| null| null| null| null| null|null| null|null| null| null| null| null|null| null| null| null| null| null| null| null| null| null| null| null| null|\n", - "|null| null|null| null|null| null| null| null| null| null|null| null|null| null| null| null| null|null| null| null| null| null| null| null| null| null| null| null| null| null|\n", - "|null| null|null| null|null| null| null| null| null| null|null| null|null| null| null| null| null|null| null| null| null| null| null| null| null| null| null| null| null| null|\n", - "|null| null|null| null|null| null| null| null| null| null|null| null|null| null| null| null| null|null| null| null| null| null| null| null| null| null| null| null| null| null|\n", - "|null| null|null| null|null| null| null| null| null| null|null| null|null| null| null| null| null|null| null| null| null| null| null| null| null| null| null| null| null| null|\n", - "|null| null|null| null|null| null| null| null| null| null|null| null|null| null| null| null| null|null| null| null| null| null| null| null| null| null| null| null| null| null|\n", - "|null| null|null| null|null| null| null| null| null| null|null| null|null| null| null| null| null|null| null| null| null| null| null| null| null| null| null| null| null| null|\n", - "|null| null|null| null|null| null| null| null| null| null|null| null|null| null| null| null| null|null| null| null| null| null| null| null| null| null| null| null| null| null|\n", - "|null| null|null| null|null| null| null| null| null| null|null| null|null| null| null| null| null|null| null| null| null| null| null| null| null| null| null| null| null| null|\n", - "|null| null|null| null|null| null| null| null| null| null|null| null|null| null| null| null| null|null| null| null| null| null| null| null| null| null| null| null| null| null|\n", - "|null| null|null| null|null| null| null| null| null| null|null| null|null| null| null| null| null|null| null| null| null| null| null| null| null| null| null| null| null| null|\n", - "|null| null|null| null|null| null| null| null| null| null|null| null|null| null| null| null| null|null| null| null| null| null| null| null| null| null| null| null| null| null|\n", - "|null| null|null| null|null| null| null| null| null| null|null| null|null| null| null| null| null|null| null| null| null| null| null| null| null| null| null| null| null| null|\n", - "|null| null|null| null|null| null| null| null| null| null|null| null|null| null| null| null| null|null| null| null| null| null| null| null| null| null| null| null| null| null|\n", - "+----+-----------+----+-----+----+------------+-----------+--------------------+------+--------+----+--------+----+--------------+--------+------------+------------+----+----------+--------+---------+--------+--------------------------+---------+---------------+-------------+-----+----------------------+----------------+------------+\n", - "only showing top 20 rows\n", - "\n" - ] - } - ], - "source": [ - "df.show()\n", - "# This comes as null which means the datatypes we gave were wrong." - ] - }, - { - "cell_type": "code", - "execution_count": 0, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 204 - }, - "colab_type": "code", - "id": "BrkVrQUsWFzk", - "outputId": "f0d8dae5-5413-4c4c-a105-b6ca5556667a" - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "+--------+-----------+-------------------+--------------------+----+--------------------+--------------------+--------------------+------+--------+----+--------+----+--------------+--------+------------+------------+----+--------------------+--------+---------+--------+\n", - "| ID|Case Number| Date| Block|IUCR| Primary Type| Description|Location Description|Arrest|Domestic|Beat|District|Ward|Community Area|FBI Code|X Coordinate|Y Coordinate|Year| Updated On|Latitude|Longitude|Location|\n", - "+--------+-----------+-------------------+--------------------+----+--------------------+--------------------+--------------------+------+--------+----+--------+----+--------------+--------+------------+------------+----+--------------------+--------+---------+--------+\n", - "|11955940| JD121140|2019-07-31 09:00:00| 026XX S HALSTED ST|1154| DECEPTIVE PRACTICE|FINANCIAL IDENTIT...| SMALL RETAIL STORE| false| false|0913| 009| 11| 60| 11| null| null|2019|01/20/2020 03:52:...| null| null| null|\n", - "|11956035| JD121288|2019-09-10 00:01:00|070XX S STONY ISL...|1582|OFFENSE INVOLVING...| CHILD PORNOGRAPHY| RESIDENCE| false| false|0332| 003| 5| 43| 17| null| null|2019|01/20/2020 03:52:...| null| null| null|\n", - "|11956045| JD121237|2019-12-15 12:00:00| 069XX S ADA ST|0610| BURGLARY| FORCIBLE ENTRY| RESIDENCE| false| false|0734| 007| 6| 67| 05| null| null|2019|01/20/2020 03:52:...| null| null| null|\n", - "|11956146| JD121344|2019-12-31 00:00:00| 016XX S THROOP ST|0610| BURGLARY| FORCIBLE ENTRY| RESIDENCE| false| false|1233| 012| 25| 31| 05| null| null|2019|01/20/2020 03:52:...| null| null| null|\n", - "|11956304| JD121233|2019-11-27 10:35:00| 002XX W GOETHE ST|0810| THEFT| OVER $500| RESIDENCE| false| false|1821| 018| 2| 8| 06| null| null|2019|01/20/2020 03:52:...| null| null| null|\n", - "+--------+-----------+-------------------+--------------------+----+--------------------+--------------------+--------------------+------+--------+----+--------+----+--------------+--------+------------+------------+----+--------------------+--------+---------+--------+\n", - "only showing top 5 rows\n", - "\n" - ] - } - ], - "source": [ - "# So let's just stick with infered schema for now, casting date to date type on the way\n", - "from pyspark.sql.functions import col, to_timestamp\n", - "df = spark.read.csv('reported-crimes.csv',header=True).withColumn('Date',to_timestamp(col('Date'),'MM/dd/yyyy hh:mm:ss a'))\n", - "df.show(5)" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "colab_type": "text", - "id": "rsD48rckdHPe" - }, - "source": [ - "\n", - "# Working with columns" - ] - }, - { - "cell_type": "code", - "execution_count": 0, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 34 - }, - "colab_type": "code", - "id": "ge9-_ygideWk", - "outputId": "9a1657dd-c7c6-4275-dffd-be200c7b8d07" - }, - "outputs": [ - { - "data": { - "text/plain": [ - "Column" - ] - }, - "execution_count": 20, - "metadata": { - "tags": [] - }, - "output_type": "execute_result" - } - ], - "source": [ - "df.Block" - ] - }, - { - "cell_type": "code", - "execution_count": 0, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 34 - }, - "colab_type": "code", - "id": "md5zaET8dsr4", - "outputId": "202211f8-aace-4955-eb5d-1ad90baa35ab" - }, - "outputs": [ - { - "data": { - "text/plain": [ - "Column" - ] - }, - "execution_count": 21, - "metadata": { - "tags": [] - }, - "output_type": "execute_result" - } - ], - "source": [ - "df['Block']" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "colab_type": "text", - "id": "YxP1su8veNde" - }, - "source": [ - "**NOTE:**\n", - "\n", - "> **We can't always use the dot notation because this will break when the column names have reserved names or attributes to the data frame class.**\n", - "\n" - ] - }, - { - "cell_type": "code", - "execution_count": 0, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 459 - }, - "colab_type": "code", - "id": "6Gkf14sHec9a", - "outputId": "ad356e2a-01d8-4562-8de5-14216be40c32" - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "+--------------------+\n", - "| Block|\n", - "+--------------------+\n", - "| 026XX S HALSTED ST|\n", - "|070XX S STONY ISL...|\n", - "| 069XX S ADA ST|\n", - "| 016XX S THROOP ST|\n", - "| 002XX W GOETHE ST|\n", - "|083XX S MUSKEGON AVE|\n", - "| 050XX W NELSON ST|\n", - "| 011XX W 83RD ST|\n", - "| 081XX W ADDISON ST|\n", - "| 031XX S PRAIRIE AVE|\n", - "|023XX N MILWAUKEE...|\n", - "| 0000X E CHESTNUT ST|\n", - "| 0000X N LATROBE AVE|\n", - "| 104XX S AVENUE J|\n", - "|025XX S CALIFORNI...|\n", - "| 021XX E 70TH ST|\n", - "|082XX S JEFFERY BLVD|\n", - "| 076XX S CICERO AVE|\n", - "| 016XX W LAKE ST|\n", - "| 018XX W DIVISION ST|\n", - "+--------------------+\n", - "only showing top 20 rows\n", - "\n" - ] - } - ], - "source": [ - "df.select(col('Block')).show()" - ] - }, - { - "cell_type": "code", - "execution_count": 0, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 459 - }, - "colab_type": "code", - "id": "05YQ7WXhiFcm", - "outputId": "cba63f1a-2071-4b77-f3f4-f6d5ed249f5e" - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "+--------------------+\n", - "| Block|\n", - "+--------------------+\n", - "| 026XX S HALSTED ST|\n", - "|070XX S STONY ISL...|\n", - "| 069XX S ADA ST|\n", - "| 016XX S THROOP ST|\n", - "| 002XX W GOETHE ST|\n", - "|083XX S MUSKEGON AVE|\n", - "| 050XX W NELSON ST|\n", - "| 011XX W 83RD ST|\n", - "| 081XX W ADDISON ST|\n", - "| 031XX S PRAIRIE AVE|\n", - "|023XX N MILWAUKEE...|\n", - "| 0000X E CHESTNUT ST|\n", - "| 0000X N LATROBE AVE|\n", - "| 104XX S AVENUE J|\n", - "|025XX S CALIFORNI...|\n", - "| 021XX E 70TH ST|\n", - "|082XX S JEFFERY BLVD|\n", - "| 076XX S CICERO AVE|\n", - "| 016XX W LAKE ST|\n", - "| 018XX W DIVISION ST|\n", - "+--------------------+\n", - "only showing top 20 rows\n", - "\n" - ] - } - ], - "source": [ - "df.select(df.Block).show()" - ] - }, - { - "cell_type": "code", - "execution_count": 0, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 459 - }, - "colab_type": "code", - "id": "NNuJ3RIqe8yY", - "outputId": "5a83e809-ae09-409c-8c1e-9861e4a2399b" - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "+--------------------+--------------------+\n", - "| Block| Description|\n", - "+--------------------+--------------------+\n", - "| 026XX S HALSTED ST|FINANCIAL IDENTIT...|\n", - "|070XX S STONY ISL...| CHILD PORNOGRAPHY|\n", - "| 069XX S ADA ST| FORCIBLE ENTRY|\n", - "| 016XX S THROOP ST| FORCIBLE ENTRY|\n", - "| 002XX W GOETHE ST| OVER $500|\n", - "|083XX S MUSKEGON AVE| $500 AND UNDER|\n", - "| 050XX W NELSON ST|HARASSMENT BY TEL...|\n", - "| 011XX W 83RD ST|SEXUAL EXPLOITATI...|\n", - "| 081XX W ADDISON ST|FINANCIAL IDENTIT...|\n", - "| 031XX S PRAIRIE AVE|ILLEGAL USE CASH ...|\n", - "|023XX N MILWAUKEE...| OVER $500|\n", - "| 0000X E CHESTNUT ST| FROM BUILDING|\n", - "| 0000X N LATROBE AVE| TO PROPERTY|\n", - "| 104XX S AVENUE J|HARASSMENT BY TEL...|\n", - "|025XX S CALIFORNI...|AGG CRIM SEX ABUS...|\n", - "| 021XX E 70TH ST|UNLAWFUL POSS OF ...|\n", - "|082XX S JEFFERY BLVD|PRO EMP HANDS NO/...|\n", - "| 076XX S CICERO AVE| $500 AND UNDER|\n", - "| 016XX W LAKE ST|DOMESTIC BATTERY ...|\n", - "| 018XX W DIVISION ST| UNLAWFUL ENTRY|\n", - "+--------------------+--------------------+\n", - "only showing top 20 rows\n", - "\n" - ] - } - ], - "source": [ - "df.select('Block','Description').show()" - ] - }, - { - "cell_type": "code", - "execution_count": 0, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 204 - }, - "colab_type": "code", - "id": "oFHUmRKZeCEV", - "outputId": "726d49bc-f384-4a31-f803-cc909151a4b2" - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "+--------+-----------+-------------------+--------------------+----+--------------------+--------------------+--------------------+------+--------+----+--------+----+--------------+--------+------------+------------+----+--------------------+--------+---------+--------+---+\n", - "| ID|Case Number| Date| Block|IUCR| Primary Type| Description|Location Description|Arrest|Domestic|Beat|District|Ward|Community Area|FBI Code|X Coordinate|Y Coordinate|Year| Updated On|Latitude|Longitude|Location|One|\n", - "+--------+-----------+-------------------+--------------------+----+--------------------+--------------------+--------------------+------+--------+----+--------+----+--------------+--------+------------+------------+----+--------------------+--------+---------+--------+---+\n", - "|11955940| JD121140|2019-07-31 09:00:00| 026XX S HALSTED ST|1154| DECEPTIVE PRACTICE|FINANCIAL IDENTIT...| SMALL RETAIL STORE| false| false|0913| 009| 11| 60| 11| null| null|2019|01/20/2020 03:52:...| null| null| null| 1|\n", - "|11956035| JD121288|2019-09-10 00:01:00|070XX S STONY ISL...|1582|OFFENSE INVOLVING...| CHILD PORNOGRAPHY| RESIDENCE| false| false|0332| 003| 5| 43| 17| null| null|2019|01/20/2020 03:52:...| null| null| null| 1|\n", - "|11956045| JD121237|2019-12-15 12:00:00| 069XX S ADA ST|0610| BURGLARY| FORCIBLE ENTRY| RESIDENCE| false| false|0734| 007| 6| 67| 05| null| null|2019|01/20/2020 03:52:...| null| null| null| 1|\n", - "|11956146| JD121344|2019-12-31 00:00:00| 016XX S THROOP ST|0610| BURGLARY| FORCIBLE ENTRY| RESIDENCE| false| false|1233| 012| 25| 31| 05| null| null|2019|01/20/2020 03:52:...| null| null| null| 1|\n", - "|11956304| JD121233|2019-11-27 10:35:00| 002XX W GOETHE ST|0810| THEFT| OVER $500| RESIDENCE| false| false|1821| 018| 2| 8| 06| null| null|2019|01/20/2020 03:52:...| null| null| null| 1|\n", - "+--------+-----------+-------------------+--------------------+----+--------------------+--------------------+--------------------+------+--------+----+--------+----+--------------+--------+------------+------------+----+--------------------+--------+---------+--------+---+\n", - "only showing top 5 rows\n", - "\n" - ] - } - ], - "source": [ - "#Adding a column in PySpark\n", - "# We are adding a column called 'One' at the end\n", - "from pyspark.sql.functions import lit\n", - "df = df.withColumn('One',lit(1))\n", - "df.show(5)" - ] - }, - { - "cell_type": "code", - "execution_count": 0, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 479 - }, - "colab_type": "code", - "id": "QJqgy6lKfk2o", - "outputId": "f690fb78-50b4-4bcc-b452-ce7308d31eef" - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "+--------+-----------+-------------------+--------------------+----+--------------------+--------------------+--------------------+------+--------+----+--------+----+--------------+--------+------------+------------+----+--------------------+------------+-------------+--------------------+----+\n", - "| ID|Case Number| Date| Block|IUCR| Primary Type| Description|Location Description|Arrest|Domestic|Beat|District|Ward|Community Area|FBI Code|X Coordinate|Y Coordinate|Year| Updated On| Latitude| Longitude| Location|Test|\n", - "+--------+-----------+-------------------+--------------------+----+--------------------+--------------------+--------------------+------+--------+----+--------+----+--------------+--------+------------+------------+----+--------------------+------------+-------------+--------------------+----+\n", - "|11955940| JD121140|2019-07-31 09:00:00| 026XX S HALSTED ST|1154| DECEPTIVE PRACTICE|FINANCIAL IDENTIT...| SMALL RETAIL STORE| false| false|0913| 009| 11| 60| 11| null| null|2019|01/20/2020 03:52:...| null| null| null| 1|\n", - "|11956035| JD121288|2019-09-10 00:01:00|070XX S STONY ISL...|1582|OFFENSE INVOLVING...| CHILD PORNOGRAPHY| RESIDENCE| false| false|0332| 003| 5| 43| 17| null| null|2019|01/20/2020 03:52:...| null| null| null| 1|\n", - "|11956045| JD121237|2019-12-15 12:00:00| 069XX S ADA ST|0610| BURGLARY| FORCIBLE ENTRY| RESIDENCE| false| false|0734| 007| 6| 67| 05| null| null|2019|01/20/2020 03:52:...| null| null| null| 1|\n", - "|11956146| JD121344|2019-12-31 00:00:00| 016XX S THROOP ST|0610| BURGLARY| FORCIBLE ENTRY| RESIDENCE| false| false|1233| 012| 25| 31| 05| null| null|2019|01/20/2020 03:52:...| null| null| null| 1|\n", - "|11956304| JD121233|2019-11-27 10:35:00| 002XX W GOETHE ST|0810| THEFT| OVER $500| RESIDENCE| false| false|1821| 018| 2| 8| 06| null| null|2019|01/20/2020 03:52:...| null| null| null| 1|\n", - "|11956126| JD121418|2019-12-23 06:00:00|083XX S MUSKEGON AVE|0820| THEFT| $500 AND UNDER| STREET| false| false|0423| 004| 7| 46| 06| null| null|2019|01/20/2020 03:52:...| null| null| null| 1|\n", - "|11956221| JD121487|2019-09-13 17:00:00| 050XX W NELSON ST|2825| OTHER OFFENSE|HARASSMENT BY TEL...| RESIDENCE| false| false|2521| 025| 31| 19| 26| null| null|2019|01/20/2020 03:52:...| null| null| null| 1|\n", - "|11956049| JD121196|2019-12-19 19:07:00| 011XX W 83RD ST|1544| SEX OFFENSE|SEXUAL EXPLOITATI...| RESIDENCE| false| false|0613| 006| 21| 71| 17| null| null|2019|01/20/2020 03:52:...| null| null| null| 1|\n", - "|11956468| JD121650|2019-06-01 08:00:00| 081XX W ADDISON ST|1153| DECEPTIVE PRACTICE|FINANCIAL IDENTIT...| APARTMENT| false| false|1631| 016| 38| 17| 11| null| null|2019|01/20/2020 03:52:...| null| null| null| 1|\n", - "|11955888| JD121051|2019-12-01 00:01:00| 031XX S PRAIRIE AVE|1152| DECEPTIVE PRACTICE|ILLEGAL USE CASH ...| RESIDENCE| false| false|0211| 002| 4| 35| 11| null| null|2019|01/20/2020 03:52:...| null| null| null| 1|\n", - "|11956127| JD121334|2019-12-14 22:00:00|023XX N MILWAUKEE...|0810| THEFT| OVER $500| OTHER| false| false|1414| 014| 1| 22| 06| null| null|2019|01/20/2020 03:52:...| null| null| null| 1|\n", - "|11956215| JD121376|2019-12-12 19:00:00| 0000X E CHESTNUT ST|0890| THEFT| FROM BUILDING| RESTAURANT| false| false|1833| 018| 42| 8| 06| null| null|2019|01/20/2020 03:52:...| null| null| null| 1|\n", - "|11956004| JD121156|2019-11-15 12:00:00| 0000X N LATROBE AVE|1310| CRIMINAL DAMAGE| TO PROPERTY| RESIDENCE| false| false|1522| 015| 28| 25| 14| null| null|2019|01/20/2020 03:52:...| null| null| null| 1|\n", - "|11956129| JD121379|2019-11-03 15:00:00| 104XX S AVENUE J|2825| OTHER OFFENSE|HARASSMENT BY TEL...| OTHER| false| false|0432| 004| 10| 52| 26| null| null|2019|01/20/2020 03:52:...| null| null| null| 1|\n", - "|11956041| JD121266|2019-12-21 14:18:00|025XX S CALIFORNI...|1752|OFFENSE INVOLVING...|AGG CRIM SEX ABUS...| APARTMENT| false| false|1033| 010| 12| 30| 17| null| null|2019|01/20/2020 03:52:...| null| null| null| 1|\n", - "|11935337| JC563784|2019-12-29 02:25:00| 021XX E 70TH ST|143A| WEAPONS VIOLATION|UNLAWFUL POSS OF ...| ALLEY| true| false|0331| 003| 5| 43| 15| 1191863| 1858976|2019|01/20/2020 03:49:...|41.768020578|-87.572284866|(41.768020578, -8...| 1|\n", - "|11927831| JC553936|2019-12-20 13:15:00|082XX S JEFFERY BLVD|0545| ASSAULT|PRO EMP HANDS NO/...| STREET| true| false|0414| 004| 8| 46| 08A| 1190954| 1850799|2019|01/20/2020 03:49:...|41.745604221|-87.575880606|(41.745604221, -8...| 1|\n", - "|11927034| JC549451|2019-12-16 19:15:00| 076XX S CICERO AVE|0820| THEFT| $500 AND UNDER| SMALL RETAIL STORE| false| false|0833| 008| 18| 65| 06| 1145727| 1853720|2019|01/20/2020 03:49:...|41.754592961|-87.741528537|(41.754592961, -8...| 1|\n", - "|11926969| JC553305|2019-12-19 21:30:00| 016XX W LAKE ST|0486| BATTERY|DOMESTIC BATTERY ...| RESIDENCE| true| false|1224| 012| 27| 28| 08B| 1165379| 1901492|2019|01/20/2020 03:49:...|41.885291047| -87.66815409|(41.885291047, -8...| 1|\n", - "|11926797| JC553193|2019-12-19 19:04:00| 018XX W DIVISION ST|0620| BURGLARY| UNLAWFUL ENTRY| RESIDENCE| true| false|1212| 012| 1| 24| 05| 1164005| 1908028|2019|01/20/2020 03:49:...|41.903255445|-87.673014935|(41.903255445, -8...| 1|\n", - "+--------+-----------+-------------------+--------------------+----+--------------------+--------------------+--------------------+------+--------+----+--------+----+--------------+--------+------------+------------+----+--------------------+------------+-------------+--------------------+----+\n", - "only showing top 20 rows\n", - "\n" - ] - } - ], - "source": [ - "#Renaming a column in PySpark\n", - "df = df.withColumnRenamed('One', 'Test')\n", - "df.show()" - ] - }, - { - "cell_type": "code", - "execution_count": 0, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 119 - }, - "colab_type": "code", - "id": "M1ek2opVfqea", - "outputId": "49be0adb-10a4-432e-e149-7fbc6cc66f9f" - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "+----+------+\n", - "|Year| count|\n", - "+----+------+\n", - "|2019|257625|\n", - "+----+------+\n", - "\n" - ] - } - ], - "source": [ - "#Group By a column in PySpark\n", - "df.groupBy('Year').count().show(5)" - ] - }, - { - "cell_type": "code", - "execution_count": 0, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 204 - }, - "colab_type": "code", - "id": "xsb9PXxpfnmh", - "outputId": "60689561-7efc-45a0-b8ae-819e7f771c8b" - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "+--------+-----------+-------------------+--------------------+----+--------------------+--------------------+--------------------+------+--------+----+--------+----+--------------+--------+------------+------------+----+--------------------+--------+---------+--------+\n", - "| ID|Case Number| Date| Block|IUCR| Primary Type| Description|Location Description|Arrest|Domestic|Beat|District|Ward|Community Area|FBI Code|X Coordinate|Y Coordinate|Year| Updated On|Latitude|Longitude|Location|\n", - "+--------+-----------+-------------------+--------------------+----+--------------------+--------------------+--------------------+------+--------+----+--------+----+--------------+--------+------------+------------+----+--------------------+--------+---------+--------+\n", - "|11955940| JD121140|2019-07-31 09:00:00| 026XX S HALSTED ST|1154| DECEPTIVE PRACTICE|FINANCIAL IDENTIT...| SMALL RETAIL STORE| false| false|0913| 009| 11| 60| 11| null| null|2019|01/20/2020 03:52:...| null| null| null|\n", - "|11956035| JD121288|2019-09-10 00:01:00|070XX S STONY ISL...|1582|OFFENSE INVOLVING...| CHILD PORNOGRAPHY| RESIDENCE| false| false|0332| 003| 5| 43| 17| null| null|2019|01/20/2020 03:52:...| null| null| null|\n", - "|11956045| JD121237|2019-12-15 12:00:00| 069XX S ADA ST|0610| BURGLARY| FORCIBLE ENTRY| RESIDENCE| false| false|0734| 007| 6| 67| 05| null| null|2019|01/20/2020 03:52:...| null| null| null|\n", - "|11956146| JD121344|2019-12-31 00:00:00| 016XX S THROOP ST|0610| BURGLARY| FORCIBLE ENTRY| RESIDENCE| false| false|1233| 012| 25| 31| 05| null| null|2019|01/20/2020 03:52:...| null| null| null|\n", - "|11956304| JD121233|2019-11-27 10:35:00| 002XX W GOETHE ST|0810| THEFT| OVER $500| RESIDENCE| false| false|1821| 018| 2| 8| 06| null| null|2019|01/20/2020 03:52:...| null| null| null|\n", - "+--------+-----------+-------------------+--------------------+----+--------------------+--------------------+--------------------+------+--------+----+--------+----+--------------+--------+------------+------------+----+--------------------+--------+---------+--------+\n", - "only showing top 5 rows\n", - "\n" - ] - } - ], - "source": [ - "#Remove columns in PySpark\n", - "df = df.drop('Test')\n", - "df.show(5)" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "colab_type": "text", - "id": "WbKK5iHwmIoV" - }, - "source": [ - "\n", - "## Working with Rows" - ] - }, - { - "cell_type": "code", - "execution_count": 0, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 479 - }, - "colab_type": "code", - "id": "YNfcjOIknA3n", - "outputId": "585cbc1a-6381-4102-d39b-f6399a87700d" - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "+--------+-----------+-------------------+--------------------+----+-------------------+--------------------+--------------------+------+--------+----+--------+----+--------------+--------+------------+------------+----+--------------------+------------+-------------+--------------------+\n", - "| ID|Case Number| Date| Block|IUCR| Primary Type| Description|Location Description|Arrest|Domestic|Beat|District|Ward|Community Area|FBI Code|X Coordinate|Y Coordinate|Year| Updated On| Latitude| Longitude| Location|\n", - "+--------+-----------+-------------------+--------------------+----+-------------------+--------------------+--------------------+------+--------+----+--------+----+--------------+--------+------------+------------+----+--------------------+------------+-------------+--------------------+\n", - "|11652463| JC221068|2019-04-11 22:30:00| 012XX W GUNNISON ST|0325| ROBBERY| VEHICULAR HIJACKING| STREET| true| false|2033| 020| 46| 3| 03| 1167294| 1932462|2019|01/20/2020 03:49:...|41.970233435|-87.660229117|(41.970233435, -8...|\n", - "|11580399| JC126732|2019-01-23 13:05:00| 033XX W FILLMORE ST|2093| NARCOTICS|FOUND SUSPECT NAR...|POLICE FACILITY/V...| true| false|1134| 011| 24| 29| 18| 1154228| 1895173|2019|01/19/2020 03:47:...|41.868180939|-87.709271389|(41.868180939, -8...|\n", - "|11953660| JD118414|2019-03-01 14:10:00| 013XX E 62ND ST|1153| DECEPTIVE PRACTICE|FINANCIAL IDENTIT...| null| false| false|0314| 003| 20| 42| 11| 1185906| 1864152|2019|01/19/2020 03:47:...|41.782366539| -87.59395659|(41.782366539, -8...|\n", - "|11955048| JD120087|2019-01-29 12:00:00| 046XX N PAULINA ST|0890| THEFT| FROM BUILDING| OTHER| false| false|1912| 019| 47| 3| 06| null| null|2019|01/19/2020 03:49:...| null| null| null|\n", - "|11953654| JD118405|2019-03-01 12:50:00| 008XX E 38TH PL|1153| DECEPTIVE PRACTICE|FINANCIAL IDENTIT...| null| false| false|0212| 002| 4| 36| 11| 1182247| 1879773|2019|01/19/2020 03:47:...| 41.82531739|-87.606887309|(41.82531739, -87...|\n", - "|11955104| JD120139|2019-05-15 16:00:00| 035XX W 71ST PL|1563| SEX OFFENSE|CRIMINAL SEXUAL A...| RESIDENCE| false| true|0831| 008| 17| 66| 17| null| null|2019|01/19/2020 03:49:...| null| null| null|\n", - "|11581568| JC126573|2019-01-23 11:00:00| 033XX W FILLMORE ST|2093| NARCOTICS|FOUND SUSPECT NAR...|POLICE FACILITY/V...| true| false|1134| 011| 24| 29| 18| 1154228| 1895173|2019|01/19/2020 03:47:...|41.868180939|-87.709271389|(41.868180939, -8...|\n", - "|11953335| JD117914|2019-01-10 21:30:00| 004XX N MC CLURG CT|1330| CRIMINAL TRESPASS| TO LAND| RESIDENCE-GARAGE| false| false|1834| 018| 42| 8| 26| 1179120| 1903342|2019|01/19/2020 03:47:...|41.890064174|-87.617638563|(41.890064174, -8...|\n", - "|11946410| JC151544|2019-02-13 20:49:00| 043XX S TROY ST|2024| NARCOTICS| POSS: HEROIN(WHITE)| RESIDENCE| true| false|0922| 009| 15| 58| 18| 1156071| 1875567|2019|01/18/2020 03:44:...|41.814342797|-87.703033557|(41.814342797, -8...|\n", - "|11677953| JC235447|2019-04-23 14:46:00|040XX W VAN BUREN ST|2014| NARCOTICS|MANU/DELIVER: HER...| VACANT LOT/LAND| true| false|1132| 011| 28| 26| 18| 1149531| 1897718|2019|01/18/2020 03:44:...|41.875257119| -87.72644908|(41.875257119, -8...|\n", - "|11672673| JC231408|2019-04-20 09:42:00| 085XX S ABERDEEN ST|1812| NARCOTICS|POSS: CANNABIS MO...| RESIDENCE| true| false|0613| 006| 21| 71| 18| 1170476| 1848176|2019|01/18/2020 03:44:...| 41.73887651|-87.650991641|(41.73887651, -87...|\n", - "|11672370| JC232204|2019-04-20 22:45:00| 011XX W 90TH ST|1812| NARCOTICS|POSS: CANNABIS MO...| APARTMENT| true| false|2222| 022| 21| 73| 18| 1170443| 1845120|2019|01/18/2020 03:44:...|41.730491134|-87.651201287|(41.730491134, -8...|\n", - "|11671344| JC235165|2019-04-23 13:17:00| 053XX W CHICAGO AVE|2017| NARCOTICS| MANU/DELIVER:CRACK| ALLEY| true| false|1524| 015| 37| 25| 18| 1140417| 1904813|2019|01/18/2020 03:44:...|41.894898651| -87.75973838|(41.894898651, -8...|\n", - "|11666457| JC228439|2019-04-17 18:34:14|054XX S CHRISTIAN...|1811| NARCOTICS|POSS: CANNABIS 30...| RESIDENCE| true| false|0822| 008| 14| 63| 18| 1154972| 1868333|2019|01/18/2020 03:44:...|41.794513753| -87.70725813|(41.794513753, -8...|\n", - "|11951328| JD115421|2019-05-01 09:00:00| 001XX E ONTARIO ST|1153| DECEPTIVE PRACTICE|FINANCIAL IDENTIT...|COMMERCIAL / BUSI...| false| false|1834| 018| 42| 8| 11| 1177462| 1904523|2019|01/17/2020 03:45:...|41.893342683| -87.62369154|(41.893342683, -8...|\n", - "|11953844| JD118256|2019-02-16 12:00:00| 062XX S KOLIN AVE|0281|CRIM SEXUAL ASSAULT| NON-AGGRAVATED| APARTMENT| false| true|0813| 008| 23| 65| 02| null| null|2019|01/17/2020 03:48:...| null| null| null|\n", - "|11950821| JC550653|2019-04-04 08:00:00| 079XX S KARLOV AVE|1154| DECEPTIVE PRACTICE|FINANCIAL IDENTIT...| RESIDENCE| false| false|0834| 008| 18| 70| 11| 1150460| 1851422|2019|01/17/2020 03:45:...|41.748196202|-87.724242888|(41.748196202, -8...|\n", - "|11951415| JD115614|2019-01-13 11:00:00|115XX S WENTWORTH...|1153| DECEPTIVE PRACTICE|FINANCIAL IDENTIT...| APARTMENT| false| false|0522| 005| 34| 53| 11| 1177005| 1828396|2019|01/17/2020 03:45:...|41.684452919|-87.627664687|(41.684452919, -8...|\n", - "|11951493| JD115459|2019-05-14 22:00:00|033XX W ARTHINGTO...|1563| SEX OFFENSE|CRIMINAL SEXUAL A...| RESIDENCE| false| true|1134| 011| 24| 29| 17| 1154345| 1895847|2019|01/17/2020 03:45:...|41.870028132|-87.708823855|(41.870028132, -8...|\n", - "|11951147| JD115168|2019-04-16 00:01:00| 005XX W MONROE ST|1120| DECEPTIVE PRACTICE| FORGERY| BANK| false| false|0121| 001| 42| 28| 10| 1172908| 1899834|2019|01/17/2020 03:45:...|41.880577917|-87.640555606|(41.880577917, -8...|\n", - "+--------+-----------+-------------------+--------------------+----+-------------------+--------------------+--------------------+------+--------+----+--------+----+--------------+--------+------------+------------+----+--------------------+------------+-------------+--------------------+\n", - "only showing top 20 rows\n", - "\n" - ] - } - ], - "source": [ - "# Filtering rows in PySpark\n", - "df.filter(col('Date')<'2019-06-01').show()" - ] - }, - { - "cell_type": "code", - "execution_count": 0, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 119 - }, - "colab_type": "code", - "id": "B1RKg1UrmBQz", - "outputId": "9653f918-dee0-47d6-a6f8-7219c51adab6" - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "+----+\n", - "|Year|\n", - "+----+\n", - "|2019|\n", - "+----+\n", - "\n" - ] - } - ], - "source": [ - "#Get Unique Rows in PySpark\n", - "df.select('Year').distinct().show()" - ] - }, - { - "cell_type": "code", - "execution_count": 0, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 479 - }, - "colab_type": "code", - "id": "4ZpeJvz0nkBI", - "outputId": "51c20b7e-67fb-49f2-c81f-955579def74d" - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "+--------+-----------+-------------------+--------------------+----+--------------------+--------------------+--------------------+------+--------+----+--------+----+--------------+--------+------------+------------+----+--------------------+------------+-------------+--------------------+\n", - "| ID|Case Number| Date| Block|IUCR| Primary Type| Description|Location Description|Arrest|Domestic|Beat|District|Ward|Community Area|FBI Code|X Coordinate|Y Coordinate|Year| Updated On| Latitude| Longitude| Location|\n", - "+--------+-----------+-------------------+--------------------+----+--------------------+--------------------+--------------------+------+--------+----+--------+----+--------------+--------+------------+------------+----+--------------------+------------+-------------+--------------------+\n", - "|11895528| JC515003|2019-01-01 00:00:00| 059XX W LELAND AVE|1153| DECEPTIVE PRACTICE|FINANCIAL IDENTIT...| null| false| false|1622| 016| 45| 15| 11| null| null|2019|11/19/2019 03:57:...| null| null| null|\n", - "|11940493| JD102895|2019-01-01 00:00:00|012XX S KOMENSKY AVE|141A| WEAPONS VIOLATION|UNLAWFUL USE HANDGUN|RESIDENCE PORCH/H...| false| false|1011| 010| 24| 29| 15| 1149582| 1893991|2019|01/08/2020 03:47:...|41.865028808|-87.726358587|(41.865028808, -8...|\n", - "|11937662| JC566620|2019-01-01 00:00:00| 019XX N HAMLIN AVE|1582|OFFENSE INVOLVING...| CHILD PORNOGRAPHY|SCHOOL, PRIVATE, ...| false| false|2535| 025| 26| 22| 17| 1150737| 1912590|2019|01/03/2020 03:56:...|41.916043916|-87.721631784|(41.916043916, -8...|\n", - "|11739188| JC326320|2019-01-01 00:00:00|010XX N RIDGEWAY AVE|1752|OFFENSE INVOLVING...|AGG CRIM SEX ABUS...| APARTMENT| false| true|1112| 011| 27| 23| 17| 1151192| 1906670|2019|12/18/2019 03:47:...|41.899789956|-87.720115618|(41.899789956, -8...|\n", - "|11911175| JC534087|2019-01-01 00:00:00| 002XX E 121ST ST|1153| DECEPTIVE PRACTICE|FINANCIAL IDENTIT...| RESIDENCE| false| false|0532| 005| 9| 53| 11| 1180018| 1824734|2019|12/07/2019 03:47:...|41.674335599|-87.616746527|(41.674335599, -8...|\n", - "|11906205| JC528532|2019-01-01 00:00:00|069XX S WOODLAWN AVE|1754|OFFENSE INVOLVING...|AGG SEX ASSLT OF ...| RESIDENCE| false| true|0321| 003| 5| 69| 02| 1185599| 1859302|2019|12/04/2019 03:49:...|41.769064952|-87.595234737|(41.769064952, -8...|\n", - "|11954981| JD119994|2019-01-01 00:00:00| 073XX W FARWELL AVE|1750|OFFENSE INVOLVING...| CHILD ABUSE| RESIDENCE| false| true|1611| 016| 41| 9| 08B| null| null|2019|01/19/2020 03:49:...| null| null| null|\n", - "|11889579| JC507635|2019-01-01 00:00:00|082XX S WENTWORTH...|1562| SEX OFFENSE|AGG CRIMINAL SEXU...| RESIDENCE| false| false|0622| 006| 21| 44| 17| 1176419| 1850436|2019|11/15/2019 03:57:...|41.744946864|-87.629149986|(41.744946864, -8...|\n", - "|11878985| JC494834|2019-01-01 00:00:00| 067XX W GRAND AVE|0810| THEFT| OVER $500| OTHER| false| false|2512| 025| 36| 18| 06| 1131156| 1915217|2019|11/06/2019 03:54:...|41.923613437|-87.793511665|(41.923613437, -8...|\n", - "|11560259| JC109129|2019-01-01 00:00:00| 049XX N ALBANY AVE|1750|OFFENSE INVOLVING...| CHILD ABUSE| APARTMENT| false| false|1713| 017| 33| 14| 08B| 1154815| 1932723|2019|10/30/2019 03:53:...| 41.9712095|-87.706108126|(41.9712095, -87....|\n", - "|11861735| JC473834|2019-01-01 00:00:00| 108XX S SANGAMON ST|1752|OFFENSE INVOLVING...|AGG CRIM SEX ABUS...| RESIDENCE| false| true|2234| 022| 34| 75| 17| 1171912| 1832941|2019|10/18/2019 04:02:...|41.697038087|-87.646175932|(41.697038087, -8...|\n", - "|11752917| JC342582|2019-01-01 00:00:00| 028XX E 76TH ST|1752|OFFENSE INVOLVING...|AGG CRIM SEX ABUS...| APARTMENT| false| true|0421| 004| 7| 43| 17| 1196262| 1855497|2019|10/15/2019 04:01:...| 41.75836602|-87.556276032|(41.75836602, -87...|\n", - "|11838300| JC441914|2019-01-01 00:00:00|031XX W ARTHINGTO...|1752|OFFENSE INVOLVING...|AGG CRIM SEX ABUS...| RESIDENCE| false| true|1134| 011| 24| 27| 17| 1155548| 1895868|2019|09/26/2019 04:16:...|41.870061657|-87.704406691|(41.870061657, -8...|\n", - "|11752915| JC342546|2019-01-01 00:00:00|063XX S STONY ISL...|1752|OFFENSE INVOLVING...|AGG CRIM SEX ABUS...| APARTMENT| false| true|0314| 003| 20| 42| 17| 1187962| 1863227|2019|09/25/2019 03:50:...|41.779779505|-87.586448286|(41.779779505, -8...|\n", - "|11739161| JC326172|2019-01-01 00:00:00| 131XX S LANGLEY AVE|1752|OFFENSE INVOLVING...|AGG CRIM SEX ABUS...| APARTMENT| false| true|0533| 005| 9| 54| 17| 1183247| 1818205|2019|09/20/2019 03:53:...| 41.65634477|-87.605129962|(41.65634477, -87...|\n", - "|11739167| JC326145|2019-01-01 00:00:00| 024XX W CARMEN AVE|1752|OFFENSE INVOLVING...|AGG CRIM SEX ABUS...| APARTMENT| false| true|2031| 020| 40| 4| 17| 1159097| 1933800|2019|09/13/2019 04:06:...|41.974077735|-87.690332986|(41.974077735, -8...|\n", - "|11808448| JC409430|2019-01-01 00:00:00|016XX E HYDE PARK...|1153| DECEPTIVE PRACTICE|FINANCIAL IDENTIT...| OTHER| false| false|0222| 002| 5| 39| 11| 1188001| 1871507|2019|08/30/2019 03:57:...|41.802499513|-87.586041451|(41.802499513, -8...|\n", - "|11803895| JC404008|2019-01-01 00:00:00| 070XX S CALUMET AVE|1153| DECEPTIVE PRACTICE|FINANCIAL IDENTIT...| RESIDENCE| false| false|0322| 003| 6| 69| 11| 1179694| 1858501|2019|08/28/2019 04:08:...|41.767003943|-87.616903894|(41.767003943, -8...|\n", - "|11785377| JC381800|2019-01-01 00:00:00| 011XX N KOSTNER AVE|1310| CRIMINAL DAMAGE| TO PROPERTY| APARTMENT| false| false|1111| 011| 37| 23| 14| 1146837| 1907072|2019|08/09/2019 04:09:...|41.900977393|-87.736101441|(41.900977393, -8...|\n", - "|11777719| JC372752|2019-01-01 00:00:00| 036XX S HALSTED ST|1120| DECEPTIVE PRACTICE| FORGERY| RESIDENCE| false| false|0915| 009| 11| 60| 10| 1171564| 1880789|2019|08/03/2019 04:02:...|41.828346576|-87.646050351|(41.828346576, -8...|\n", - "+--------+-----------+-------------------+--------------------+----+--------------------+--------------------+--------------------+------+--------+----+--------+----+--------------+--------+------------+------------+----+--------------------+------------+-------------+--------------------+\n", - "only showing top 20 rows\n", - "\n" - ] - } - ], - "source": [ - "# Sort Rows in PySpark\n", - "df.orderBy('Date').show()" - ] - }, - { - "cell_type": "code", - "execution_count": 0, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 34 - }, - "colab_type": "code", - "id": "bCZIzfYmnx--", - "outputId": "06e38cb1-cccc-44a5-f005-0a485b98a7fe" - }, - "outputs": [ - { - "data": { - "text/plain": [ - "10" - ] - }, - "execution_count": 33, - "metadata": { - "tags": [] - }, - "output_type": "execute_result" - } - ], - "source": [ - "# Append rows in PySpark.\n", - "one_day = spark.read.csv('reported-crimes.csv',header=True).withColumn('Date',to_timestamp(col('Date'),'MM/dd/yyyy hh:mm:ss a')).filter(col('Date')==lit('2019-07-30'))\n", - "df.filter(col('Date')==lit('2019-07-30')).count()" - ] - }, - { - "cell_type": "code", - "execution_count": 0, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 34 - }, - "colab_type": "code", - "id": "k9CZlZ4Got_e", - "outputId": "db825a82-8e9a-4ea1-eff7-2f370676d6b5" - }, - "outputs": [ - { - "data": { - "text/plain": [ - "20" - ] - }, - "execution_count": 34, - "metadata": { - "tags": [] - }, - "output_type": "execute_result" - } - ], - "source": [ - "df.union(one_day).filter(col('Date')==lit('2019-07-30')).count()" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "colab_type": "text", - "id": "1pfPzVOFqC_8" - }, - "source": [ - "**Result:**\n", - "\n", - "> As you can see here, there were 6 crimes commited on 2019-07-30, and after union, there's 12 records.\n", - "\n" - ] - }, - { - "cell_type": "code", - "execution_count": 0, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 289 - }, - "colab_type": "code", - "id": "6tVJeAY_plob", - "outputId": "bd7b6a38-55f1-4e21-897b-04fb1a2f4661" - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "+-------------------+-----+\n", - "| Primary Type|count|\n", - "+-------------------+-----+\n", - "| THEFT|62137|\n", - "| BATTERY|49425|\n", - "| CRIMINAL DAMAGE|26638|\n", - "| ASSAULT|20584|\n", - "| DECEPTIVE PRACTICE|17674|\n", - "| OTHER OFFENSE|16543|\n", - "| NARCOTICS|13946|\n", - "| BURGLARY| 9590|\n", - "|MOTOR VEHICLE THEFT| 8968|\n", - "| ROBBERY| 7985|\n", - "+-------------------+-----+\n", - "only showing top 10 rows\n", - "\n" - ] - } - ], - "source": [ - "# Top 10 number of reported crimes by Primary Type, in descending order of Occurence\n", - "df.groupBy(\"Primary Type\").count().orderBy('count',ascending=False).show(10)" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "colab_type": "text", - "id": "xOQPOt19q_he" - }, - "source": [ - "\n", - "# Hands-on Questions 🤚 !" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "colab_type": "text", - "id": "uk7x_PCarIff" - }, - "source": [ - "**What percentage of reported crimes resulted in an arrest?**" - ] - }, - { - "cell_type": "code", - "execution_count": 0, - "metadata": { - "colab": {}, - "colab_type": "code", - "id": "zZkkAz-srcMf" - }, - "outputs": [], - "source": [ - "# Answer" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "colab_type": "text", - "id": "9Qzp-Vb-ra6w" - }, - "source": [ - "**What are the top 3 locations for reported crimes?**" - ] - }, - { - "cell_type": "code", - "execution_count": 0, - "metadata": { - "colab": {}, - "colab_type": "code", - "id": "qI6srWS4rYwv" - }, - "outputs": [], - "source": [ - "# Answer" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "colab_type": "text", - "id": "aHjILb1DriuX" - }, - "source": [ - "\n", - "# Functions" - ] - }, - { - "cell_type": "code", - "execution_count": 0, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 54 - }, - "colab_type": "code", - "id": "x3vlC7ZerlKb", - "outputId": "05314019-519c-4fb9-c27a-dec7e1a5251a" - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "['Column', 'DataFrame', 'DataType', 'PandasUDFType', 'PythonEvalType', 'SparkContext', 'StringType', 'UserDefinedFunction', '__all__', '__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__spec__', '_binary_mathfunctions', '_collect_list_doc', '_collect_set_doc', '_create_binary_mathfunction', '_create_column_from_literal', '_create_function', '_create_udf', '_create_window_function', '_functions', '_functions_1_4', '_functions_1_6', '_functions_2_1', '_functions_2_4', '_functions_deprecated', '_lit_doc', '_message', '_string_functions', '_test', '_to_java_column', '_to_seq', '_window_functions', '_wrap_deprecated_function', 'abs', 'acos', 'add_months', 'approxCountDistinct', 'approx_count_distinct', 'array', 'array_contains', 'array_distinct', 'array_except', 'array_intersect', 'array_join', 'array_max', 'array_min', 'array_position', 'array_remove', 'array_repeat', 'array_sort', 'array_union', 'arrays_overlap', 'arrays_zip', 'asc', 'asc_nulls_first', 'asc_nulls_last', 'ascii', 'asin', 'atan', 'atan2', 'avg', 'base64', 'basestring', 'bin', 'bitwiseNOT', 'blacklist', 'broadcast', 'bround', 'cbrt', 'ceil', 'coalesce', 'col', 'collect_list', 'collect_set', 'column', 'concat', 'concat_ws', 'conv', 'corr', 'cos', 'cosh', 'count', 'countDistinct', 'covar_pop', 'covar_samp', 'crc32', 'create_map', 'cume_dist', 'current_date', 'current_timestamp', 'date_add', 'date_format', 'date_sub', 'date_trunc', 'datediff', 'dayofmonth', 'dayofweek', 'dayofyear', 'decode', 'degrees', 'dense_rank', 'desc', 'desc_nulls_first', 'desc_nulls_last', 'element_at', 'encode', 'exp', 'explode', 'explode_outer', 'expm1', 'expr', 'factorial', 'first', 'flatten', 'floor', 'format_number', 'format_string', 'from_json', 'from_unixtime', 'from_utc_timestamp', 'functools', 'get_json_object', 'greatest', 'grouping', 'grouping_id', 'hash', 'hex', 'hour', 'hypot', 'ignore_unicode_prefix', 'initcap', 'input_file_name', 'instr', 'isnan', 'isnull', 'json_tuple', 'kurtosis', 'lag', 'last', 'last_day', 'lead', 'least', 'length', 'levenshtein', 'lit', 'locate', 'log', 'log10', 'log1p', 'log2', 'lower', 'lpad', 'ltrim', 'map_concat', 'map_from_arrays', 'map_from_entries', 'map_keys', 'map_values', 'max', 'md5', 'mean', 'min', 'minute', 'monotonically_increasing_id', 'month', 'months_between', 'nanvl', 'next_day', 'ntile', 'pandas_udf', 'percent_rank', 'posexplode', 'posexplode_outer', 'pow', 'quarter', 'radians', 'rand', 'randn', 'rank', 'regexp_extract', 'regexp_replace', 'repeat', 'reverse', 'rint', 'round', 'row_number', 'rpad', 'rtrim', 'schema_of_json', 'second', 'sequence', 'sha1', 'sha2', 'shiftLeft', 'shiftRight', 'shiftRightUnsigned', 'shuffle', 'signum', 'sin', 'since', 'sinh', 'size', 'skewness', 'slice', 'sort_array', 'soundex', 'spark_partition_id', 'split', 'sqrt', 'stddev', 'stddev_pop', 'stddev_samp', 'struct', 'substring', 'substring_index', 'sum', 'sumDistinct', 'sys', 'tan', 'tanh', 'toDegrees', 'toRadians', 'to_date', 'to_json', 'to_timestamp', 'to_utc_timestamp', 'translate', 'trim', 'trunc', 'udf', 'unbase64', 'unhex', 'unix_timestamp', 'upper', 'var_pop', 'var_samp', 'variance', 'warnings', 'weekofyear', 'when', 'window', 'year']\n" - ] - } - ], - "source": [ - "# Functions available in PySpark\n", - "from pyspark.sql import functions\n", - "print(dir(functions))" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "colab_type": "text", - "id": "PIKigra7A34e" - }, - "source": [ - "\n", - "## String Functions" - ] - }, - { - "cell_type": "code", - "execution_count": 0, - "metadata": { - "colab": {}, - "colab_type": "code", - "id": "63QDccSjBqC4" - }, - "outputs": [], - "source": [ - "# Loading the data\n", - "from pyspark.sql.functions import col\n", - "df = spark.read.csv('reported-crimes.csv',header=True).withColumn('Date',to_timestamp(col('Date'),'MM/dd/yyyy hh:mm:ss a'))" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "colab_type": "text", - "id": "LiXWN8DUA9x6" - }, - "source": [ - "**Display the Primary Type column in lower and upper characters, and the first 4 characters of the column**" - ] - }, - { - "cell_type": "code", - "execution_count": 0, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 459 - }, - "colab_type": "code", - "id": "52Gh9c99BZFr", - "outputId": "88e24118-eb24-41e7-8856-bcf914264e63" - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Help on function substring in module pyspark.sql.functions:\n", - "\n", - "substring(str, pos, len)\n", - " Substring starts at `pos` and is of length `len` when str is String type or\n", - " returns the slice of byte array that starts at `pos` in byte and is of length `len`\n", - " when str is Binary type.\n", - " \n", - " .. note:: The position is not zero based, but 1 based index.\n", - " \n", - " >>> df = spark.createDataFrame([('abcd',)], ['s',])\n", - " >>> df.select(substring(df.s, 1, 2).alias('s')).collect()\n", - " [Row(s='ab')]\n", - " \n", - " .. versionadded:: 1.5\n", - "\n", - "+--------------------+--------------------+-----------------------------+\n", - "| lower(Primary Type)| upper(Primary Type)|substring(Primary Type, 1, 4)|\n", - "+--------------------+--------------------+-----------------------------+\n", - "| deceptive practice| DECEPTIVE PRACTICE| DECE|\n", - "|offense involving...|OFFENSE INVOLVING...| OFFE|\n", - "| burglary| BURGLARY| BURG|\n", - "| burglary| BURGLARY| BURG|\n", - "| theft| THEFT| THEF|\n", - "+--------------------+--------------------+-----------------------------+\n", - "only showing top 5 rows\n", - "\n" - ] - } - ], - "source": [ - "from pyspark.sql.functions import col,lower, upper, substring\n", - "help(substring)\n", - "df.select(lower(col('Primary Type')),upper(col('Primary Type')),substring(col('Primary Type'),1,4)).show(5)" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "colab_type": "text", - "id": "ldtA0wk9BMkT" - }, - "source": [ - "\n", - "## Numeric functions" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "colab_type": "text", - "id": "kmz4G5LVBOs6" - }, - "source": [ - "**Show the oldest date and the most recent date**" - ] - }, - { - "cell_type": "code", - "execution_count": 0, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 119 - }, - "colab_type": "code", - "id": "wBDDH-YpBbdk", - "outputId": "988c8eb7-a03b-4926-f4d1-2d4e60b13e28" - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "+-------------------+-------------------+\n", - "| min(Date)| max(Date)|\n", - "+-------------------+-------------------+\n", - "|2019-01-01 00:00:00|2019-12-31 23:55:00|\n", - "+-------------------+-------------------+\n", - "\n" - ] - } - ], - "source": [ - "from pyspark.sql.functions import min, max\n", - "df.select(min(col('Date')), max(col('Date'))).show()" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "colab_type": "text", - "id": "KQ6Ul9HGCwC3" - }, - "source": [ - "\n", - "## Date" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "colab_type": "text", - "id": "9WFrADdkBPsX" - }, - "source": [ - "**What is 3 days earlier that the oldest date and 3 days later than the most recent date?**" - ] - }, - { - "cell_type": "code", - "execution_count": 0, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 119 - }, - "colab_type": "code", - "id": "YrNmD5umAx__", - "outputId": "e0ff823f-cdc1-4f2b-933e-f716b23d1a74" - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "+----------------------+----------------------+\n", - "|date_add(max(Date), 3)|date_sub(min(Date), 3)|\n", - "+----------------------+----------------------+\n", - "| 2020-01-03| 2018-12-29|\n", - "+----------------------+----------------------+\n", - "\n" - ] - } - ], - "source": [ - "from pyspark.sql.functions import date_add, date_sub\n", - "df.select(date_add(max(col('Date')),3), date_sub(min(col('Date')),3)).show()" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "colab_type": "text", - "id": "sY6PstyLDp6P" - }, - "source": [ - "\n", - "# Working with Dates" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "colab_type": "text", - "id": "s1jmBN2qFHyk" - }, - "source": [ - "> [PySpark follows SimpleDateFormat table of Java. Click here to view the docs.](https://docs.oracle.com/javase/7/docs/api/java/text/SimpleDateFormat.html)" - ] - }, - { - "cell_type": "code", - "execution_count": 0, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 119 - }, - "colab_type": "code", - "id": "sCTeI_JvDCsH", - "outputId": "786f1816-5e7e-4745-a46d-103be8a80294" - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "+-------------------+\n", - "| Christmas|\n", - "+-------------------+\n", - "|2019-12-25 13:30:00|\n", - "+-------------------+\n", - "\n" - ] - } - ], - "source": [ - "from pyspark.sql.functions import to_date, to_timestamp, lit\n", - "df = spark.createDataFrame([('2019-12-25 13:30:00',)], ['Christmas'])\n", - "df.show()" - ] - }, - { - "cell_type": "code", - "execution_count": 0, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 119 - }, - "colab_type": "code", - "id": "ZH8ja1eHEW8x", - "outputId": "4947fd01-5043-40c4-fcef-c2335af911ea" - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "+-------------------------------------------+------------------------------------------------+\n", - "|to_date(`Christmas`, 'yyyy-MM-dd HH:mm:ss')|to_timestamp(`Christmas`, 'yyyy-MM-dd HH:mm:ss')|\n", - "+-------------------------------------------+------------------------------------------------+\n", - "| 2019-12-25| 2019-12-25 13:30:00|\n", - "+-------------------------------------------+------------------------------------------------+\n", - "\n" - ] - } - ], - "source": [ - "df.select(to_date(col('Christmas'),'yyyy-MM-dd HH:mm:ss'), to_timestamp(col('Christmas'),'yyyy-MM-dd HH:mm:ss')).show()" - ] - }, - { - "cell_type": "code", - "execution_count": 0, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 119 - }, - "colab_type": "code", - "id": "7g9m_8PPErI1", - "outputId": "941eed2f-7002-4311-cc71-61913e8767ba" - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "+--------------------------------------------+-------------------------------------------------+\n", - "|to_date(`Christmas`, 'dd/MMM/yyyy HH:mm:ss')|to_timestamp(`Christmas`, 'dd/MMM/yyyy HH:mm:ss')|\n", - "+--------------------------------------------+-------------------------------------------------+\n", - "| 2019-12-25| 2019-12-25 13:30:00|\n", - "+--------------------------------------------+-------------------------------------------------+\n", - "\n" - ] - } - ], - "source": [ - "df = spark.createDataFrame([('25/Dec/2019 13:30:00',)], ['Christmas'])\n", - "df.select(to_date(col('Christmas'),'dd/MMM/yyyy HH:mm:ss'), to_timestamp(col('Christmas'),'dd/MMM/yyyy HH:mm:ss')).show()" - ] - }, - { - "cell_type": "code", - "execution_count": 0, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 323 - }, - "colab_type": "code", - "id": "x26Ls5SzE9qJ", - "outputId": "5f499b28-b27e-4de2-ac61-df11fabd5aeb" - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "+--------------------+\n", - "| Christmas|\n", - "+--------------------+\n", - "|12/25/2019 01:30:...|\n", - "+--------------------+\n", - "\n", - "+----------------------+\n", - "|Christmas |\n", - "+----------------------+\n", - "|12/25/2019 01:30:00 PM|\n", - "+----------------------+\n", - "\n", - "+----------------------------------------------+---------------------------------------------------+\n", - "|to_date(`Christmas`, 'MM/dd/yyyy hh:mm:ss aa')|to_timestamp(`Christmas`, 'MM/dd/yyyy hh:mm:ss aa')|\n", - "+----------------------------------------------+---------------------------------------------------+\n", - "| 2019-12-25| 2019-12-25 13:30:00|\n", - "+----------------------------------------------+---------------------------------------------------+\n", - "\n" - ] - } - ], - "source": [ - "df = spark.createDataFrame([('12/25/2019 01:30:00 PM',)], ['Christmas'])\n", - "df.show(1)\n", - "df.show(1, truncate = False)\n", - "df.select(to_date(col('Christmas'),'MM/dd/yyyy hh:mm:ss aa'), to_timestamp(col('Christmas'),'MM/dd/yyyy hh:mm:ss aa')).show()" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "colab_type": "text", - "id": "7OZElEvcGOD1" - }, - "source": [ - "\n", - "# Working with joins" - ] - }, - { - "cell_type": "code", - "execution_count": 0, - "metadata": { - "colab": {}, - "colab_type": "code", - "id": "UJBC7r3JFyCL" - }, - "outputs": [], - "source": [ - "# Loading the data\n", - "from pyspark.sql.functions import col\n", - "df = spark.read.csv('reported-crimes.csv',header=True).withColumn('Date',to_timestamp(col('Date'),'MM/dd/yyyy hh:mm:ss a'))" - ] - }, - { - "cell_type": "code", - "execution_count": 0, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 204 - }, - "colab_type": "code", - "id": "-_aNpIqwIERe", - "outputId": "7e26e2fe-fabe-441c-c09c-0f14e64377d9" - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "--2020-01-22 04:30:02-- https://data.cityofchicago.org/api/views/z8bn-74gv/rows.csv?accessType=DOWNLOAD\n", - "Resolving data.cityofchicago.org (data.cityofchicago.org)... 52.206.140.205, 52.206.68.26, 52.206.140.199\n", - "Connecting to data.cityofchicago.org (data.cityofchicago.org)|52.206.140.205|:443... connected.\n", - "HTTP request sent, awaiting response... 200 OK\n", - "Length: unspecified [text/csv]\n", - "Saving to: ‘police-station.csv’\n", - "\n", - "\r", - "police-station.csv [<=> ] 0 --.-KB/s \r", - "police-station.csv [ <=> ] 5.57K --.-KB/s in 0s \n", - "\n", - "2020-01-22 04:30:03 (562 MB/s) - ‘police-station.csv’ saved [5699]\n", - "\n" - ] - } - ], - "source": [ - "# Dowloading police station data\n", - "!wget -O police-station.csv https://data.cityofchicago.org/api/views/z8bn-74gv/rows.csv?accessType=DOWNLOAD" - ] - }, - { - "cell_type": "code", - "execution_count": 0, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 459 - }, - "colab_type": "code", - "id": "HOQjNSrdIO1n", - "outputId": "d7e8ad6f-750a-4ca9-e0d2-ea624a19b3f2" - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "+--------------------+-----------------+--------------------+-------+-----+-----+--------------------+------------+------------+------------+------------+------------+-----------+------------+--------------------+\n", - "| DISTRICT| DISTRICT NAME| ADDRESS| CITY|STATE| ZIP| WEBSITE| PHONE| FAX| TTY|X COORDINATE|Y COORDINATE| LATITUDE| LONGITUDE| LOCATION|\n", - "+--------------------+-----------------+--------------------+-------+-----+-----+--------------------+------------+------------+------------+------------+------------+-----------+------------+--------------------+\n", - "| 1| Central| 1718 S State St|Chicago| IL|60616|http://home.chica...|312-745-4290|312-745-3694|312-745-3693| 1176569.052| 1891771.704|41.85837259|-87.62735617|(41.8583725929, -...|\n", - "| 6| Gresham| 7808 S Halsted St|Chicago| IL|60620|http://home.chica...|312-745-3617|312-745-3649|312-745-3639| 1172283.013| 1853022.646|41.75213684|-87.64422891|(41.7521368378, -...|\n", - "| 11| Harrison| 3151 W Harrison St|Chicago| IL|60612|http://home.chica...|312-746-8386|312-746-4281|312-746-5151| 1155244.069| 1897148.755|41.87358229|-87.70548813|(41.8735822883, -...|\n", - "| 16| Jefferson Park|5151 N Milwaukee Ave|Chicago| IL|60630|http://home.chica...|312-742-4480|312-742-4421|312-742-4423| 1138480.758| 1933660.473|41.97409445|-87.76614884|(41.9740944511, -...|\n", - "| Headquarters| Headquarters| 3510 S Michigan Ave|Chicago| IL|60653|http://home.chica...| null| null| null| 1177731.401| 1881697.404|41.83070169|-87.62339535|(41.8307016873, -...|\n", - "| 24| Rogers Park| 6464 N Clark St|Chicago| IL|60626|http://home.chica...|312-744-5907|312-744-6928|312-744-7603| 1164193.588| 1943199.401|41.99976348|-87.67132429|(41.9997634842, -...|\n", - "| 2| Wentworth|5101 S Wentworth Ave|Chicago| IL|60609|http://home.chica...|312-747-8366|312-747-5396|312-747-6656| 1175864.837| 1871153.753|41.80181109|-87.63056018|(41.8018110912, -...|\n", - "| 7| Englewood| 1438 W 63rd St|Chicago| IL|60636|http://home.chica...|312-747-8223|312-747-6558|312-747-6652| 1167659.235| 1863005.522|41.77963154|-87.66088702|(41.7796315359, -...|\n", - "| 25| Grand Central| 5555 W Grand Ave|Chicago| IL|60639|http://home.chica...|312-746-8605|312-746-4353|312-746-8383| 1138770.871| 1913442.439|41.91860889|-87.76557448|(41.9186088912, -...|\n", - "| 10| Ogden| 3315 W Ogden Ave|Chicago| IL|60623|http://home.chica...|312-747-7511|312-747-7429|312-747-7471| 1154500.753| 1890985.501|41.85668453|-87.70838196|(41.8566845327, -...|\n", - "| 15| Austin| 5701 W Madison St|Chicago| IL|60644|http://home.chica...|312-743-1440|312-743-1366|312-743-1485| 1138148.815| 1899399.078|41.88008346|-87.76819989|(41.8800834614, -...|\n", - "| 3| Grand Crossing|7040 S Cottage Gr...|Chicago| IL|60637|http://home.chica...|312-747-8201|312-747-5479|312-747-9168| 1182739.183| 1858317.732|41.76643089|-87.60574786|(41.7664308925, -...|\n", - "| 14| Shakespeare|2150 N California...|Chicago| IL|60647|http://home.chica...|312-744-8250|312-744-2422|312-744-8260| 1157304.426| 1914481.521|41.92110332|-87.69745182|(41.9211033246, -...|\n", - "| 8| Chicago Lawn| 3420 W 63rd St|Chicago| IL|60629|http://home.chica...|312-747-8730|312-747-8545|312-747-8116| 1154575.242| 1862672.049|41.77898719|-87.70886382|(41.778987189, -8...|\n", - "| 4| South Chicago| 2255 E 103rd St|Chicago| IL|60617|http://home.chica...|312-747-7581|312-747-5276|312-747-9169| 1193131.299| 1837090.265|41.70793329|-87.56834912|(41.7079332906, -...|\n", - "| 20| Lincoln| 5400 N Lincoln Ave|Chicago| IL|60625|http://home.chica...|312-742-8714|312-742-8803|312-742-8841| 1158399.146| 1935788.826|41.97954951|-87.69284451|(41.9795495131, -...|\n", - "| 18| Near North| 1160 N Larrabee St|Chicago| IL|60610|http://home.chica...|312-742-5870|312-742-5771|312-742-5773| 1172080.029| 1908086.527|41.90324165|-87.64335214|(41.9032416531, -...|\n", - "| 12| Near West|1412 S Blue Islan...| null| null| null| null| null| null| null| null| null| null| null| null|\n", - "|\",Chicago,IL,6060...| -87.6569725149)\"| null| null| null| null| null| null| null| null| null| null| null| null| null|\n", - "| 9| Deering| 3120 S Halsted St|Chicago| IL|60608|http://home.chica...|312-747-8227|312-747-5329|312-747-9172| 1171440.24| 1884085.224|41.83739443|-87.64640771|(41.8373944311, -...|\n", - "+--------------------+-----------------+--------------------+-------+-----+-----+--------------------+------------+------------+------------+------------+------------+-----------+------------+--------------------+\n", - "only showing top 20 rows\n", - "\n" - ] - } - ], - "source": [ - "ps = spark.read.csv(\"police-station.csv\", header=True)\n", - "ps.show()" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "colab_type": "text", - "id": "hY05yWIEJIdb" - }, - "source": [ - "**The reported crimes dataset has only the district number. Add the district name by joining with the police station dataset.**" - ] - }, - { - "cell_type": "code", - "execution_count": 0, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 34 - }, - "colab_type": "code", - "id": "QePrc33RI25U", - "outputId": "59d62a27-b7bf-454a-bc5f-b9214cdeb7ad" - }, - "outputs": [ - { - "data": { - "text/plain": [ - "257625" - ] - }, - "execution_count": 48, - "metadata": { - "tags": [] - }, - "output_type": "execute_result" - } - ], - "source": [ - "# Caching the crimes dataset to speed things up, and then, since spark does lazy evaluation, gonna run an action to make it evaluated.\n", - "df.cache()\n", - "df.count()" - ] - }, - { - "cell_type": "code", - "execution_count": 0, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 901 - }, - "colab_type": "code", - "id": "lSsxX7qoJinQ", - "outputId": "cf8ced90-8228-4297-a750-ee09b00cbe7d" - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "+------------+\n", - "| DISTRICT|\n", - "+------------+\n", - "| 7|\n", - "| 15|\n", - "| 11|\n", - "| 3|\n", - "| 8|\n", - "| 22|\n", - "| 16|\n", - "| 5|\n", - "| 18|\n", - "| 17|\n", - "| 6|\n", - "| 19|\n", - "| 25|\n", - "|Headquarters|\n", - "| 24|\n", - "| 9|\n", - "| 1|\n", - "| 20|\n", - "| 10|\n", - "| 4|\n", - "+------------+\n", - "only showing top 20 rows\n", - "\n", - "+--------+\n", - "|District|\n", - "+--------+\n", - "| 009|\n", - "| 012|\n", - "| 024|\n", - "| 031|\n", - "| 015|\n", - "| 006|\n", - "| 019|\n", - "| 020|\n", - "| 011|\n", - "| 025|\n", - "| 003|\n", - "| 005|\n", - "| 016|\n", - "| 018|\n", - "| 008|\n", - "| 022|\n", - "| 001|\n", - "| 014|\n", - "| 010|\n", - "| 004|\n", - "+--------+\n", - "only showing top 20 rows\n", - "\n" - ] - } - ], - "source": [ - "ps.select(col('DISTRICT')).distinct().show()\n", - "df.select(col('District')).distinct().show()" - ] - }, - { - "cell_type": "code", - "execution_count": 0, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 459 - }, - "colab_type": "code", - "id": "7keEhyBMJsFT", - "outputId": "38229d80-ea97-4213-e727-356ae9e021b4" - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "+---------------+\n", - "|Format_district|\n", - "+---------------+\n", - "| 001|\n", - "| 006|\n", - "| 011|\n", - "| 016|\n", - "| Hea|\n", - "| 024|\n", - "| 002|\n", - "| 007|\n", - "| 025|\n", - "| 010|\n", - "| 015|\n", - "| 003|\n", - "| 014|\n", - "| 008|\n", - "| 004|\n", - "| 020|\n", - "| 018|\n", - "| 012|\n", - "| \",C|\n", - "| 009|\n", - "+---------------+\n", - "only showing top 20 rows\n", - "\n" - ] - } - ], - "source": [ - "# Transfomring crime data to remove 0 from beginning, inroder to match the data\n", - "from pyspark.sql.functions import lpad\n", - "ps = ps.withColumn('Format_district',lpad(col('DISTRICT'),3,'0'))\n", - "ps.select(col('Format_district')).show()" - ] - }, - { - "cell_type": "code", - "execution_count": 0, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 459 - }, - "colab_type": "code", - "id": "U7Py4EYyKJTN", - "outputId": "8cb9a1eb-78d0-4dfa-c4fb-9256eb9467ad" - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "+--------+-----------+-------------------+--------------------+----+--------------------+--------------------+--------------------+------+--------+----+--------+----+--------------+--------+----+--------------------+--------+--------------+---------------+\n", - "| ID|Case Number| Date| Block|IUCR| Primary Type| Description|Location Description|Arrest|Domestic|Beat|District|Ward|Community Area|FBI Code|Year| Updated On|DISTRICT| DISTRICT NAME|Format_district|\n", - "+--------+-----------+-------------------+--------------------+----+--------------------+--------------------+--------------------+------+--------+----+--------+----+--------------+--------+----+--------------------+--------+--------------+---------------+\n", - "|11955940| JD121140|2019-07-31 09:00:00| 026XX S HALSTED ST|1154| DECEPTIVE PRACTICE|FINANCIAL IDENTIT...| SMALL RETAIL STORE| false| false|0913| 009| 11| 60| 11|2019|01/20/2020 03:52:...| 9| Deering| 009|\n", - "|11956035| JD121288|2019-09-10 00:01:00|070XX S STONY ISL...|1582|OFFENSE INVOLVING...| CHILD PORNOGRAPHY| RESIDENCE| false| false|0332| 003| 5| 43| 17|2019|01/20/2020 03:52:...| 3|Grand Crossing| 003|\n", - "|11956045| JD121237|2019-12-15 12:00:00| 069XX S ADA ST|0610| BURGLARY| FORCIBLE ENTRY| RESIDENCE| false| false|0734| 007| 6| 67| 05|2019|01/20/2020 03:52:...| 7| Englewood| 007|\n", - "|11956146| JD121344|2019-12-31 00:00:00| 016XX S THROOP ST|0610| BURGLARY| FORCIBLE ENTRY| RESIDENCE| false| false|1233| 012| 25| 31| 05|2019|01/20/2020 03:52:...| 12| Near West| 012|\n", - "|11956304| JD121233|2019-11-27 10:35:00| 002XX W GOETHE ST|0810| THEFT| OVER $500| RESIDENCE| false| false|1821| 018| 2| 8| 06|2019|01/20/2020 03:52:...| 18| Near North| 018|\n", - "|11956126| JD121418|2019-12-23 06:00:00|083XX S MUSKEGON AVE|0820| THEFT| $500 AND UNDER| STREET| false| false|0423| 004| 7| 46| 06|2019|01/20/2020 03:52:...| 4| South Chicago| 004|\n", - "|11956221| JD121487|2019-09-13 17:00:00| 050XX W NELSON ST|2825| OTHER OFFENSE|HARASSMENT BY TEL...| RESIDENCE| false| false|2521| 025| 31| 19| 26|2019|01/20/2020 03:52:...| 25| Grand Central| 025|\n", - "|11956049| JD121196|2019-12-19 19:07:00| 011XX W 83RD ST|1544| SEX OFFENSE|SEXUAL EXPLOITATI...| RESIDENCE| false| false|0613| 006| 21| 71| 17|2019|01/20/2020 03:52:...| 6| Gresham| 006|\n", - "|11956468| JD121650|2019-06-01 08:00:00| 081XX W ADDISON ST|1153| DECEPTIVE PRACTICE|FINANCIAL IDENTIT...| APARTMENT| false| false|1631| 016| 38| 17| 11|2019|01/20/2020 03:52:...| 16|Jefferson Park| 016|\n", - "|11955888| JD121051|2019-12-01 00:01:00| 031XX S PRAIRIE AVE|1152| DECEPTIVE PRACTICE|ILLEGAL USE CASH ...| RESIDENCE| false| false|0211| 002| 4| 35| 11|2019|01/20/2020 03:52:...| 2| Wentworth| 002|\n", - "|11956127| JD121334|2019-12-14 22:00:00|023XX N MILWAUKEE...|0810| THEFT| OVER $500| OTHER| false| false|1414| 014| 1| 22| 06|2019|01/20/2020 03:52:...| 14| Shakespeare| 014|\n", - "|11956215| JD121376|2019-12-12 19:00:00| 0000X E CHESTNUT ST|0890| THEFT| FROM BUILDING| RESTAURANT| false| false|1833| 018| 42| 8| 06|2019|01/20/2020 03:52:...| 18| Near North| 018|\n", - "|11956004| JD121156|2019-11-15 12:00:00| 0000X N LATROBE AVE|1310| CRIMINAL DAMAGE| TO PROPERTY| RESIDENCE| false| false|1522| 015| 28| 25| 14|2019|01/20/2020 03:52:...| 15| Austin| 015|\n", - "|11956129| JD121379|2019-11-03 15:00:00| 104XX S AVENUE J|2825| OTHER OFFENSE|HARASSMENT BY TEL...| OTHER| false| false|0432| 004| 10| 52| 26|2019|01/20/2020 03:52:...| 4| South Chicago| 004|\n", - "|11956041| JD121266|2019-12-21 14:18:00|025XX S CALIFORNI...|1752|OFFENSE INVOLVING...|AGG CRIM SEX ABUS...| APARTMENT| false| false|1033| 010| 12| 30| 17|2019|01/20/2020 03:52:...| 10| Ogden| 010|\n", - "|11935337| JC563784|2019-12-29 02:25:00| 021XX E 70TH ST|143A| WEAPONS VIOLATION|UNLAWFUL POSS OF ...| ALLEY| true| false|0331| 003| 5| 43| 15|2019|01/20/2020 03:49:...| 3|Grand Crossing| 003|\n", - "|11927831| JC553936|2019-12-20 13:15:00|082XX S JEFFERY BLVD|0545| ASSAULT|PRO EMP HANDS NO/...| STREET| true| false|0414| 004| 8| 46| 08A|2019|01/20/2020 03:49:...| 4| South Chicago| 004|\n", - "|11927034| JC549451|2019-12-16 19:15:00| 076XX S CICERO AVE|0820| THEFT| $500 AND UNDER| SMALL RETAIL STORE| false| false|0833| 008| 18| 65| 06|2019|01/20/2020 03:49:...| 8| Chicago Lawn| 008|\n", - "|11926969| JC553305|2019-12-19 21:30:00| 016XX W LAKE ST|0486| BATTERY|DOMESTIC BATTERY ...| RESIDENCE| true| false|1224| 012| 27| 28| 08B|2019|01/20/2020 03:49:...| 12| Near West| 012|\n", - "|11926797| JC553193|2019-12-19 19:04:00| 018XX W DIVISION ST|0620| BURGLARY| UNLAWFUL ENTRY| RESIDENCE| true| false|1212| 012| 1| 24| 05|2019|01/20/2020 03:49:...| 12| Near West| 012|\n", - "+--------+-----------+-------------------+--------------------+----+--------------------+--------------------+--------------------+------+--------+----+--------+----+--------------+--------+----+--------------------+--------+--------------+---------------+\n", - "only showing top 20 rows\n", - "\n" - ] - } - ], - "source": [ - "# Executing the join and deleting some column so that we don't get too much data\n", - "df.join(ps, df.District == ps.Format_district, 'left_outer').drop(\n", - " 'ADDRESS',\n", - " 'CITY',\n", - " 'STATE',\n", - " 'ZIP',\n", - " 'WEBSITE',\n", - " 'PHONE',\n", - " 'FAX',\n", - " 'TTY',\n", - " 'X COORDINATE',\n", - " 'Y COORDINATE',\n", - " 'LATITUDE',\n", - " 'LONGITUDE',\n", - " 'LOCATION').show()" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "colab_type": "text", - "id": "EEEB2TVqL4Ie" - }, - "source": [ - "\n", - "# Hands-on again!" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "colab_type": "text", - "id": "50gdkX-_MEMv" - }, - "source": [ - "**What is the most frequently reported non-criminal activity?**" - ] - }, - { - "cell_type": "code", - "execution_count": 0, - "metadata": { - "colab": {}, - "colab_type": "code", - "id": "pbjuihaOL-fM" - }, - "outputs": [], - "source": [] - }, - { - "cell_type": "markdown", - "metadata": { - "colab_type": "text", - "id": "UdMXUsjKME2y" - }, - "source": [ - "**Find the day of the week with the most reported crime?**" - ] - }, - { - "cell_type": "code", - "execution_count": 0, - "metadata": { - "colab": {}, - "colab_type": "code", - "id": "7Sdq1ZwRMN0p" - }, - "outputs": [], - "source": [] - }, - { - "cell_type": "markdown", - "metadata": { - "colab_type": "text", - "id": "REpHzg_nNctg" - }, - "source": [ - "**Using a bar chart, plot which day of the week has the most number of reported crime.**" - ] - }, - { - "cell_type": "code", - "execution_count": 0, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 51 - }, - "colab_type": "code", - "id": "j5olv2qkNfPa", - "outputId": "c6c14f40-9415-474b-fbc2-fabfb3596f30" - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "['Sun', 'Mon', 'Thu', 'Sat', 'Wed', 'Tue', 'Fri']\n", - "[20036, 21041, 20701, 21786, 20652, 21331, 22147]\n" - ] - } - ], - "source": [ - "from pyspark.sql.functions import date_format, col\n", - "dow = [x[0] for x in df.groupBy(date_format(col('Date'),'E')).count().collect()]\n", - "print(dow)\n", - "cnt = [x[1] for x in df.groupBy(date_format(col('Date'),'E')).count().collect()]\n", - "print(cnt)" - ] - }, - { - "cell_type": "code", - "execution_count": 0, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 204 - }, - "colab_type": "code", - "id": "gzOCDOqFOOc6", - "outputId": "860dba1a-8e8d-4629-bc80-43d60a59becb" - }, - "outputs": [ - { - "data": { - "text/html": [ - "
    \n", - "\n", - "\n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - "
    Day_of_weekCount
    0Sun20036
    1Mon21041
    2Thu20701
    3Sat21786
    4Wed20652
    \n", - "
    " - ], - "text/plain": [ - " Day_of_week Count\n", - "0 Sun 20036\n", - "1 Mon 21041\n", - "2 Thu 20701\n", - "3 Sat 21786\n", - "4 Wed 20652" - ] - }, - "execution_count": 203, - "metadata": { - "tags": [] - }, - "output_type": "execute_result" - } - ], - "source": [ - "import pandas as pd\n", - "import matplotlib.pyplot as plt\n", - "\n", - "cp = pd.DataFrame({'Day_of_week':dow, 'Count':cnt})\n", - "cp.head()" - ] - }, - { - "cell_type": "code", - "execution_count": 0, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 325 - }, - "colab_type": "code", - "id": "J4pxrZeSOvIS", - "outputId": "c1511ea1-19b9-4603-c10c-3e6c85fcbae6" - }, - "outputs": [ - { - "data": { - "text/plain": [ - "Text(0.5, 1.0, 'No.of reported crimes per day')" - ] - }, - "execution_count": 208, - "metadata": { - "tags": [] - }, - "output_type": "execute_result" - }, - { - "data": { - "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZUAAAEjCAYAAAD6yJxTAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3WmYHVW59vH/TQgkkEAgBAQCBBFQ\nZkIYREFEBUQmEREcCMgB8YjKERD0VYLiACjiQQWMjHFgUKaoIHI4RCYRkjAPSg4EEiAhhClECJA8\n74daTSqb7t3V3bW7utL377rq6l2rpmfvdPbTtdaqtRQRmJmZlWGZqgMwM7Olh5OKmZmVxknFzMxK\n46RiZmalcVIxM7PSOKmYmVlpnFSsT5G0hqSbJc2TdEbV8bSCpJD0rhLOs5Okf5YRU11IOlTSrVXH\nYR1zUrFOSZou6VlJK+bK/kPSpBZc7kjgOWCliDi2Befvkb70pRYRt0TExlXHYZbnpGJFDQC+2gvX\nWQ94KAo+lStpQIvjyV9r2d66Vmf6UixlWRrfU3/kpGJF/Qg4TtKw9jZK2lHSXZJeSj937OhEHe0r\n6SJgLPB1Sa9I+nA7x14k6RxJ10qaD3xQ0vKSfizpSUmzJZ0raXDafxdJMyV9U9Jz6a7rM7nzrSxp\ngqQ5kp6Q9C1Jy6Rth0q6TdKZkuYClwHnAu9N8b2Y9uvw+mn78ZKekfS0pM83+5AlrSrpwrTvC5Ku\nbngfJ0iaBVzYVpY7dnq61n2S5ks6P1UnXpeqE/9H0iq5/XeQdLukFyXdK2mX3LZDJT2Wjns8/5k1\nxHuypD9IuiztO1XSlrnta0m6In2+j0v6SjvH/kbSy8Ch7Zx/uKSJkl6WdCewQcP2/5Y0I22fImmn\nVP4OSf+WNDy37+gUx8Bm/wbWQxHhxUvTBZgOfBi4EvheKvsPYFJ6vSrwAvA5YFng4LQ+vJ1zNd0X\nuKjtGh3EchHwEvA+sj+KBgFnAhPTuYcCfwR+mPbfBXgT+AmwPPABYD6wcdo+AbgmHTcK+BdweNp2\naDr2yynWwans1oaYml1/D2A2sBmwIvA7IIB3dfD+/kyWvFYBBgIfaHgfp6X3MTiVzWz4d7oDWANY\nG3gWmApsnT6n/wXGpX3XBuYCe6bP8SNpfUSK8+XcZ7QmsGkH8Z4MvAEckOI9Dng8vV4GmAKcBCwH\nvBN4DNi94dj90r6D2zn/pcDlKabNgKfynz/wWWB4+vc5FpgFDErbrgW+2PDv9LOq/z8t7UvlAXjp\n+wuLk8pmZF/oI1gyqXwOuLPhmL8Dh7Zzrqb7UiypTMitiyxJbJArey/weHrd9mW8Ym775cC3yar0\nXgc2yW37Qu59HQo82XD9Qxu+1Dq7/gXAqbltG9FBUklf3ouAVdrZtkuKdVBDWWNS+Uxu/QrgnNz6\nl4Gr0+sTgF83XON6sjvFFYEXgU+090XfcMzJwB259WWAZ4CdgO3b+fy+AVyYO/bmJuceQJZ03p0r\n+wENSb3hmBeALdPrTwG35c41C9iu6v9PS/viOkwrLCIekPQn4ETg4dymtYAnGnZ/guyv4UZd2bcj\nM3KvRwArAFMktZWJ7EukzQsRMb/hemsBq5H9Rf1Ew7Z8LPlrtaez669F9td6/vwdWQd4PiJe6GD7\nnIh4rZN4Zudev9rO+pD0ej3gk5L2zm0fCNwUEfMlfYrsruN8SbcBx0bEIx1c863PKCIWpSq5tciS\n51pt1YTJAOCW9o5txwiyO5D8Pkt8fpKOAw7PXW8lsn9XyO5Az5W0PrAx8FJE3NnkelYCt6lYV40D\njmDJL96nyb6k8tYlq6po1JV9O5JvxH+O7Mty04gYlpaVI2JIbp9VlOu5lq73dDr2jYZ4GmNp7DDQ\nuN7Z9Z8hSxb583dkBrBqR+1W7Vy7J2aQ3akMyy0rRsSpABFxfUR8hOzu6RHgV03O9db7S+1RI8k+\n3xlkd2z5awyNiD0Lvqc5ZHeZ7X5+qf3k68CBZHd3w8jupJXew2tkd6WfJbtD/nWTa1lJnFSsSyJi\nGlmd/1dyxdcCG0n6tKRl01+5mwB/aucUXdm3SDyLyL7wzpS0OoCktSXt3rDrdyQtl76I9gJ+HxEL\nyb50vi9pqKT1gK8Bv2lyydnASEnLFbz+5cChkjaRtAJZUu7ovTwDXAecLWkVSQMl7dyVz6MLfgPs\nLWl3SQMkDUoN/yNT4/6+KREvAF4hq5bryDaS9lfWe+uYdMwdwJ3AvNS5YHC6zmaSti0SYPr3uRI4\nWdIKkjYhq55rM5Qs6cwBlpV0EtmdSt4EsirLfXBS6RVOKtYd3yWrdwcgIuaSfVEfS9bY+3Vgr4h4\nDiD1hjq3yL7ddAIwDbgj9SL6H7LqjjazyOranwZ+CxyVq8r5MlmbyGPArWQN6Rc0udb/Ag8CsyS1\nxdzh9SPiOuCn6bhp6WcznyO7e3qErKH9mE7275aImAHsC3yT7Et5BnA82XfCMmTJ9WngebLODV9s\ncrpryNov2jpg7B8Rb6SksBewFVnj/XPAecDKXQj1aLIqu1lk7WkX5rZdD/yFrHPFE8BrNFSnRcRt\nZAlxakQ0q3q0kijCk3TZ0it1k/1NRIysOpalkaSTyTodfLbqWDoi6X+B30XEeVXH0h+4od7Mllqp\nqm002V2Z9QJXf5nZUknSxWRVkcdExLyq4+kvXP1lZmal8Z2KmZmVxknFzMxK0+8a6ldbbbUYNWpU\n1WGYmdXKlClTnouIEZ3t1++SyqhRo5g8eXLVYZiZ1YqkQs/5uPrLzMxK46RiZmalcVIxM7PS9Ls2\nFTOzjrzxxhvMnDmT117rbIaBpdegQYMYOXIkAwd2b4JMJxUzs2TmzJkMHTqUUaNGkZsfp9+ICObO\nncvMmTNZf/31u3UOV3+ZmSWvvfYaw4cP75cJBUASw4cP79GdmpOKmVlOf00obXr6/p1UzMz6kFmz\nZnHQQQexwQYbsM0227Dnnnvyr3/9q7TzT5o0idtvv7208zVym0pnWv1Xiwf0NOu7yv7/38n/94jg\n4x//OGPHjuXSSy8F4N5772X27NlstNFGpYQwadIkhgwZwo477ljK+Rr5TsXMrI+46aabGDhwIEcd\nddRbZVtuuSXvf//7Of7449lss83YfPPNueyyy4AsQey1115v7Xv00Udz0UUXAdnoIePGjWP06NFs\nvvnmPPLII0yfPp1zzz2XM888k6222opbbrml9PfgOxUzsz7igQceYJtttnlb+ZVXXsk999zDvffe\ny3PPPce2227Lzjvv3On5VlttNaZOncrZZ5/Nj3/8Y8477zyOOuoohgwZwnHHHdeKt+A7FTOzvu7W\nW2/l4IMPZsCAAayxxhp84AMf4K677ur0uP333x+AbbbZhunTp7c4yoyTytJMau1iZqXadNNNmTJl\nSuH9l112WRYtWvTWemNX4OWXXx6AAQMG8Oabb5YTZCecVMzM+ohdd92VBQsWMH78+LfK7rvvPoYN\nG8Zll13GwoULmTNnDjfffDPbbbcd6623Hg899BALFizgxRdf5MYbb+z0GkOHDmXevNbNruw2FTOz\nPkISV111FccccwynnXYagwYNYtSoUfz0pz/llVdeYcstt0QSp59+Ou94xzsAOPDAA9lss81Yf/31\n2XrrrTu9xt57780BBxzANddcw89+9jN22mmnct9Df5ujfsyYMdGl+VTq3KW4zrGbVeDhhx/mPe95\nT9VhVK69z0HSlIgY09mxvlOxvstJ0ax23KZiZmalcVIxM7PSuPrLrFVcfVdLEdGvB5XsaTu771TM\nzJJBgwYxd+7cHn+x1lXbfCqDBg3q9jl8p2JmlowcOZKZM2cyZ86cqkOpTNvMj93lpGJm7at79V03\n4h8IFJ7vsJ/ezXTG1V9mZlYaJxUzMytNy5KKpHUk3STpIUkPSvpqKl9V0g2SHk0/V0nlknSWpGmS\n7pM0OneusWn/RyWNzZVvI+n+dMxZ6s9dNszM+oBW3qm8CRwbEZsAOwBfkrQJcCJwY0RsCNyY1gE+\nCmyYliOBcyBLQsA4YHtgO2BcWyJK+xyRO26PFr4fMzPrRMuSSkQ8ExFT0+t5wMPA2sC+wMVpt4uB\n/dLrfYEJkbkDGCZpTWB34IaIeD4iXgBuAPZI21aKiDsi6/83IXcuMzOrQK/0/pI0Ctga+AewRkQ8\nkzbNAtZIr9cGZuQOm5nKmpXPbKe8vesfSXb3w7rrrtv9N2Jm1ltq2vuu5Q31koYAVwDHRMTL+W3p\nDqPl/fIiYnxEjImIMSNGjGj15czM+q2WJhVJA8kSym8j4spUPDtVXZF+PpvKnwLWyR0+MpU1Kx/Z\nTrmZmVWklb2/BJwPPBwRP8ltmgi09eAaC1yTKz8k9QLbAXgpVZNdD+wmaZXUQL8bcH3a9rKkHdK1\nDsmdy8zMKtDKNpX3AZ8D7pd0Tyr7JnAqcLmkw4EngAPTtmuBPYFpwL+BwwAi4nlJpwB3pf2+GxHP\np9f/CVwEDAauS4uZmVXEMz92pqaNZUC9YwfH3xnH35zjb66L8Red+dFP1JuZWWmcVMzMrDROKmZm\nVhonFTMzK42TipmZlcZJxczMSuOkYmZmpek0qUg6XdJKkgZKulHSHEmf7Y3gzMysXorcqeyWBoLc\nC5gOvAs4vpVBmZlZPRVJKm1DuXwM+H1EvNTCeMzMrMaKjP31J0mPAK8CX5Q0AnittWGZmVkddXqn\nEhEnAjsCYyLiDbLBHvdtdWBmZlY/RRrqVyAbDficVLQW0OmgYmZm1v8UaVO5EHid7G4Fsomwvtey\niMzMrLaKJJUNIuJ04A2AiPg30OIxmc3MrI6KJJXXJQ0mzSUvaQNgQUujMjOzWirS+2sc8BdgHUm/\nJZvR8dBWBmVmZvXUaVKJiBskTQV2IKv2+mpEPNfyyMzMrHaKjv21NjAAWA7YWdL+rQvJzMzqqtM7\nFUkXAFsADwKLUnEAV7YwLjMzq6EibSo7RMQmLY/EzMxqr0j1198lOamYmVmnitypTCBLLLPIuhIL\niIjYoqWRmZlZ7RRJKucDnwPuZ3GbipmZ2dsUSSpzImJiyyMxM7PaK5JU7pb0O+CP5J6kjwj3/jIz\nsyUUSSqDyZLJbrkydyk2M7O3KfJE/WG9EYiZmdVfh0lF0tcj4nRJPyMNJpkXEV9paWRmZlY7ze5U\nHk4/J/dGIGZmVn8dJpWI+KOkAcDmEXFcL8ZkZmY11fSJ+ohYSDbUvZmZWaeK9P66R9JE4PfA/LZC\ndyk2M7NGRZLKIGAusGuuzF2Kzczsbdyl2MzMStPpKMWSLpY0LLe+SppjxczMbAlFhr7fIiJebFuJ\niBeArVsXkpmZ1VWRpLKMpFXaViStSsEZIyU9K+mBXNnJkp6SdE9a9sxt+4akaZL+KWn3XPkeqWya\npBNz5etL+kcqv0zSckXesJmZtU6RpHIG2Xwqp0g6BbgdOL3AcRcBe7RTfmZEbJWWawHSJGAHAZum\nY86WNCA9J/ML4KPAJsDBuQnDTkvnehfwAnB4gZjMzKyFOk0qETEB2B+YnZb9I+LXBY67GXi+YBz7\nApdGxIKIeByYBmyXlmkR8VhEvA5cCuwrSWS90f6Qjr8Y2K/gtczMrEWKdCkmIh4CHirpmkdLOoRs\n+JdjUxvN2sAduX1mpjKAGQ3l2wPDgRcj4s129jczs4oUqf4q0znABsBWwDNkVWstJ+lISZMlTZ4z\nZ05vXNLMrF/q1aQSEbMjYmFELAJ+RVa9BfAUsE5u15GprKPyucAwScs2lHd03fERMSYixowYMaKc\nN2NmZm/Tq0lF0pq51Y8DbT3DJgIHSVpe0vrAhsCdwF3Ahqmn13JkjfkTIyKAm4AD0vFjgWt64z2Y\nmVnHms2nMo925lFpExErNTuxpEuAXYDVJM0ExgG7SNoqnXc68IV0rgclXU7WbvMm8KU0mCWSjgau\nBwYAF0TEg+kSJwCXSvoecDdwfmdv1szMWkvZH/1Ndsi6ET8D/BoQ8BlgzYg4qfXhlW/MmDExeXIX\npoiRWhcMQCeff4/UOXZw/J1x/M05/ua6GL+kKRExprP9ilR/7RMRZ0fEvIh4OSLOIesCbGZmtoQi\nSWW+pM+khxGXkfQZckPgm5mZtSmSVD4NHMjihx8/mcrMzMyWUGTo++m4usvMzAooMvT9RpJubBsY\nUtIWkr7V+tDMzKxuilR//Qr4BvAGQETcR/a8iJmZ2RKKJJUVIuLOhrI3293TzMz6tSJJ5TlJG5Ae\nhJR0ANlzK2ZmZksoMkrxl4DxwLslPQU8TvYApJmZ2RKKJJWIiA9LWhFYJiLmpfG5zMzMllCk+usK\ngIiYHxHzUtkfmuxvZmb9VLMBJd9NNr3vypL2z21aCRjU6sDMzKx+mlV/bQzsBQwD9s6VzwOOaGVQ\nZmZWTx0mlYi4RtKfgBMi4ge9GJOZmdVU0zaVNKfJfr0Ui5mZ1VyR3l+3Sfo5cBm50YkjYmrLojIz\ns1oqklS2Sj+/mysLYNfywzEzszorMkrxB3sjEDMzq78ioxSvLOknkian5QxJK/dGcGZmVi9FHn68\ngKwb8YFpeRm4sJVBmZlZPRVpU9kgIj6RW/+OpHtaFZCZmdVXkTuVVyW9v21F0vuAV1sXkpmZ1VWR\nO5UvAhendhQBzwNjWxqVmZnVUpHeX/cAW0paKa2/3PKozMyslor0/hou6SxgEnCTpP+WNLzlkZmZ\nWe0UaVO5FJgDfAI4IL2+rJVBmZlZPRVpU1kzIk7JrX9P0qdaFZCZmdVXkTuVv0o6SNIyaTkQuL7V\ngZmZWf0USSpHAL8DXgcWkFWHfUHSPElutDczs7cU6f01tDcCMTOz+ivS+0uSPivp22l9HUnbtT40\nMzOrmyLVX2cD7wU+ndZfAX7RsojMzKy2ivT+2j4iRku6GyAiXpC0XIvjMjOzGipyp/KGpAFkE3Mh\naQSwqKVRmZlZLRVJKmcBVwGrS/o+cCvwg5ZGZWZmtVSk99dvJU0BPkQ2oOR+EfFwyyMzM7PaaZpU\nUrXXgxHxbuCR3gnJzMzqqmn1V0QsBP4pad1eisfMzGqsSJvKKsCDkm6UNLFt6ewgSRdIelbSA7my\nVSXdIOnR9HOVVC5JZ0maJuk+SaNzx4xN+z8qaWyufBtJ96djzpKkrr11MzMrW5Euxd/u5rkvAn4O\nTMiVnQjcGBGnSjoxrZ8AfBTYMC3bA+cA20taFRgHjCHrfTZF0sSIeCHtcwTwD+BaYA/gum7GamZm\nJSjSUP+37pw4Im6WNKqheF9gl/T6YrI5Wk5I5RMiIoA7JA2TtGba94aIeB5A0g3AHpImAStFxB2p\nfAKwH04qZmaVKlL9VaY1IuKZ9HoWsEZ6vTYwI7ffzFTWrHxmO+VmZlah3k4qb0l3JdEb15J0pKTJ\nkibPmTOnNy5pZtYvdZhUJN2Yfp5W4vVmp2ot0s9nU/lTwDq5/UamsmblI9spb1dEjI+IMRExZsSI\nET1+E2Zm1r5mdyprStoR2EfS1pJG55duXm8i0NaDayxwTa78kNQLbAfgpVRNdj2wm6RVUk+x3YDr\n07aXJe2Qen0dkjuXmZlVpFlD/UlkPb9GAj9p2BbArs1OLOkSsob21STNJOvFdSpwuaTDgSeAA9Pu\n1wJ7AtOAfwOHAUTE85JOAe5K+323rdEe+E+yHmaDyRro3UhvZlYxZU0bTXaQvt0wR32tjRkzJiZP\nnlz8gFY//tLJ598jdY4dHH9nHH9zjr+5LsYvaUpEjOlsvyJdik+RtA+wcyqaFBF/6lI0ZmbWLxSZ\n+fGHwFeBh9LyVUkepdjMzN6myBP1HwO2iohFAJIuBu4GvtnKwMzMrH6KPqcyLPd65VYEYmZm9Vfk\nTuWHwN2SbiKbT2VnsjG7zMzMllCkof6SNNbWtqnohIiY1dKozMyslorcqZAeNux0uHszM+vfKhv7\ny8zMlj5OKmZmVpqmSUXSAEmem97MzArxHPVmZlaaIg31bXPU3wnMbyuMiH1aFpWZmdVSK+eoNzOz\nfqbQHPWS1gM2jIj/kbQCMKD1oZmZWd0UGVDyCOAPwC9T0drA1a0MyszM6qlIl+IvAe8DXgaIiEeB\n1VsZlJmZ1VORpLIgIl5vW5G0LNnMj2ZmZksoklT+JumbwGBJHwF+D/yxtWGZmVkdFUkqJwJzgPuB\nL5DNJ/+tVgZlZmb1VKT316I0Mdc/yKq9/hmdTWxvZmb9UqdJRdLHgHOB/yObT2V9SV+IiOtaHZyZ\nmdVLkYcfzwA+GBHTACRtAPwZcFIxM7MlFGlTmdeWUJLHgHktisfMzGqswzsVSfunl5MlXQtcTtam\n8kngrl6IzczMaqZZ9dfeudezgQ+k13OAwS2LyMzMaqvDpBIRh/VmIGZmVn9Fen+tD3wZGJXf30Pf\nm5lZoyK9v64Gzid7in5Ra8MxM7M6K5JUXouIs1oeiZmZ1V6RpPLfksYBfwUWtBVGxNSWRWVmZrVU\nJKlsDnwO2JXF1V+R1s3MzN5SJKl8Enhnfvh7MzOz9hR5ov4BYFirAzEzs/orcqcyDHhE0l0s2abi\nLsVmZraEIkllXMujMDOzpUKR+VT+1huBmJlZ/RV5on4ei+ekXw4YCMyPiJVaGZiZmdVPkTuVoW2v\nJQnYF9ihlUGZmVk9Fen99ZbIXA3s3pOLSpou6X5J90ianMpWlXSDpEfTz1VSuSSdJWmapPskjc6d\nZ2za/1FJY3sSk5mZ9VyR6q/9c6vLAGOA10q49gcj4rnc+onAjRFxqqQT0/oJwEeBDdOyPXAOsL2k\nVck6EYwhq56bImliRLxQQmxmZtYNRXp/5edVeROYTlYFVrZ9gV3S64uBSWRJZV9gQkQEcIekYZLW\nTPveEBHPA0i6AdgDuKQFsZmZWQFF2lRaMa9KAH+VFMAvI2I8sEZEPJO2zwLWSK/XBmbkjp2Zyjoq\nNzOzijSbTvikJsdFRJzSg+u+PyKekrQ6cIOkRxpPnhJOKSQdCRwJsO6665Z1WjMza9CsoX5+OwvA\n4WTVUt0WEU+ln88CVwHbAbNTtRbp57Np96eAdXKHj0xlHZW3d73xETEmIsaMGDGiJ6GbmVkTHSaV\niDijbQHGk81LfxhwKfDO7l5Q0oqShra9BnYjG19sItDWg2sscE16PRE4JPUC2wF4KVWTXQ/sJmmV\n1FNst1RmZmYVadqmknpYfQ34DFnj+egSeletAVyVPfLCssDvIuIvaWyxyyUdDjwBHJj2vxbYE5gG\n/JsssRERz0s6Bbgr7ffdtkZ7MzOrRrM2lR8B+5PdpWweEa+UccGIeAzYsp3yucCH2ikP4EsdnOsC\n4IIy4jIzs55r1qZyLLAW8C3gaUkvp2WepJd7JzwzM6uTDu9UIqJLT9ubmZk5cZiZWWmcVMzMrDRO\nKmZmVhonFTMzK42TipmZlcZJxczMSuOkYmZmpXFSMTOz0jipmJlZaZxUzMysNE4qZmZWGicVMzMr\njZOKmZmVxknFzMxK46RiZmalcVIxM7PSOKmYmVlpnFTMzKw0TipmZlYaJxUzMyuNk4qZmZXGScXM\nzErjpGJmZqVxUjEzs9I4qZiZWWmcVMzMrDROKmZmVhonFTMzK42TipmZlcZJxczMSuOkYmZmpXFS\nMTOz0jipmJlZaZxUzMysNE4qZmZWGicVMzMrTe2TiqQ9JP1T0jRJJ1Ydj5lZf1brpCJpAPAL4KPA\nJsDBkjapNiozs/6r1kkF2A6YFhGPRcTrwKXAvhXHZGbWby1bdQA9tDYwI7c+E9i+cSdJRwJHptVX\nJP2zhTGtBjxXeG+pdZF0XZ1jB8dfNcdfrVbHv16RneqeVAqJiPHA+N64lqTJETGmN65VtjrHDo6/\nao6/Wn0l/rpXfz0FrJNbH5nKzMysAnVPKncBG0paX9JywEHAxIpjMjPrt2pd/RURb0o6GrgeGABc\nEBEPVhxWr1SztUidYwfHXzXHX60+Eb8iouoYzMxsKVH36i8zM+tDnFTMzKw0TipmZlYaJ5V+TtLy\nRcr6IknvknS9pHvT+haSvlF1XP2FpMclPda4VB2XVcsN9T0g6fKIOFDS/UD+gxQQEbFFRaEVJmlq\nRIzurKwvkjQJ+Cbwi4jYWpKAByJi02oj65o0ht0a5HpjRsST1UVUjKThudVBwCeBVSPipIpCKkTS\n15ptj4if9FYsPdUXf3dq3aW4D/hq+rlXpVF0g6R3kA1zM1jS1mSJEGAlYIXKAuuaFSPidqXhJiIi\nJL1RcUxdIunLwDhgNrAoFQfQ5/8giYi5DUU/lTQF6NNJBRiafm4MbMviZ9v2Bu6sJKJu6Ku/O04q\nPRARz6S/FC6KiA9WHU8X7Q4cSjYKQf4vs3lkf/3XwVxJ65PuEiXtB8yqNqQu+yqwcTtf0H2epPzd\n7DLAGGrwnRIR3wGQdDMwOiLmpfWTgT9XGFpX9cnfnT7/C9DXRcRCSYskrRwRL1UdT1ERcTFwsaRP\nRMQVVcfTTUcD5wPvlvQE8AxwcLUhddkMoDa/Nw3OyL1+E5gOHFhNKN2yBvB6bv31VFYXffJ3x0ml\nHK8A90u6AZjfVhgRX6kupGIi4gpJHwM2JasXbyv/bnVRFRMR04BdJa1M1j74YtUxdcNjwCRJfwYW\ntBXWoV6/hnfnjSYAd0q6Kq3vB1xcYTxd1Sd/d5xUynFlWmpH0rlkbSgfBM4DDqAm9cqSvtmwDkBE\n/KCSgLrnybQsl5baSL0EPwGMYsmG4j7/BwlARHxf0nXATqnosIi4u8qYuqhP/u6491cPSFq36p4W\nPSXpvojYIvdzCHBdROzU6cEVk3RCbnUQ8DHgwYg4rKKQui197kTEK1XHUpSkv5BVv0wBFraVR8QZ\nHR7Ux0h6P7BhRFwoaQQwJCIerzquOvOdSs9cDYwGkHRFRHyi4ni649X089+S1gKeB9asMJ7CIuK0\n/Lqk04C/VBROt0jaDPg1sGpafw44pA8MjFrEyIjYo+ogukvSOLLOBRsDFwIDgd8A76syrqIk3cSS\njzIAEBG7VhDOW5xUeiY/ddo7K4uiZ/4kaRhwOtlfnJBVg9XR8mS92epkPPC1iLgJQNIuwK+AHasM\nqqDbJW0eEfdXHUg3fRzYGpgKEBFPSxra/JA+5bjc60FkVZFvVhTLW5xUeiY6eN3nSdoWmBERp6T1\nIcD9wCPAmVXG1hlJy6ZpD+6TG2doAAAF+0lEQVRm8ec+gOwOq07tKZA9a3NT20pETJK0YpUBdUbS\nA2TPRSwLHJaeol9AjR76TV5Pzza1dUnv0597o4iY0lB0m6TK20OdVHpmS0kvk/1nGpxew+L/XCtV\nF1qnfgl8GEDSzsCpwJeBrcj+ej6gutA6dSdZtWM+xjeBWRGxoP1D+qzHJH2brAoM4LNkvXr6srXJ\nfk/q7nJJvwSGSToC+DzZXWItSFo1t9r2nNDKFYXzFieVHoiIAVXH0AMDIuL59PpTwPj0vMoVku6p\nMK4iBBAR/1d1ICX4PPAdFvcevCWV9WWPR8QTVQfRXZKOAW4HfkrW6/FlsnaVkyLihipj66IpLL5T\nb3tO6PDKokmcVPqvAW3VSMCHgCNz2/r678WIZuM3Vd1Pvysi4gWgzz/P1GD1mn/+I8kSyrvJqnxv\nI0syjdVJfVKu6nr9tD6WrD1lOvBQhaEBff/Lw1rnEuBvqbfRq2R/ISPpXfTBp3QbDACGsGRHiVqR\nNLHZ9ojYp7di6YZaf/4RcRyApOXIqox2BA4Dxkt6MSI2qTK+Ahqrrn9IH6q6dlLpp9KDXzeSNW7/\nNRY/sLQM2S9oX/ZMXR6wa+K9ZMNsXAL8g3p9QS8Nnz/AYLIBVFdOy9Nkdy59XZ+uunZS6cci4o52\nyv5VRSxdVKcv4I68A/gI2VhlnyYbyPCSmjyfUuvPX9J4smGJ5pEl9NuBn6SqyDro01XXnqTL6uhD\nVQfQUxGxMCL+EhFjgR2AaWTjOB1dcWhF1P3zX5fsmaZZwFPATKBO48a1VV1fQx+suvYwLWYVSWNn\nfYzsbmUU2bweF0TEU1XG1R+kCd02JWtP2RHYjGw0ib9HxLgqYytC0g4srrqen8o2IhtmZmqlsTmp\nmPU+SRPIvsiuBS6NiAcqDqlfkjSSbFiWHckm2xseEcOqjarenFTMKiBpEYunSWhvKuq+/OBsrUn6\nCovvUN4ga1NpW+6PiEVNDrdOVN6oY9YfRYTbM6szCvg98F8R8UzFsSx1fKdiZmal8V9LZmZWGicV\nMzMrjZOKWQNJCyXdI+lBSfdKOlZSS/+vSPpRut6PWnydUWnoerOWcEO92du9GhFbAUhaHfgd2XAe\nrXx+4Uhg1YhY2OmeZn2Y71TMmoiIZ8m+8I9WZpSkWyRNTcuOkD13Imm/tuMk/VbSvvlzpeN/JOkB\nSfdL+lQqn0g2QOOUtrLcMfdLGpaOnSvpkNz1PiJpQDrnXZLuk/SF3LHH58q/0/jeJL1T0t1p1Fuz\nUvhOxawTEfGYpAHA6sCzwEci4jVJG5INmTEGOB/4L+BqSSuTPQMxtuFU+5ONJLslsBpwl6SbI2If\nSa+03R01uI3s4bwnyCbv2gmYQDYg5RfJ5s94KSK2TU/o3ybpr8CGadmO7NmXiWlE2ycBJG0MXAoc\nGhH39vxTMss4qZh1zUDg55K2AhYCGwFExN8knS1pBNncFlekAf/y3k82aORCYLakvwHbkg3P0pFb\ngJ3Jkso5wJGS1gZeiIj5knYDtpDUNtz5ymTJZLe03J3Kh6TyJ4ERwDXA/hFR+fwbtnRxUjHrhKR3\nkiWQZ8naVWaT3W0sA7yW23UC2XTAB5HNz1GGm4EvkQ2C+P+Aj5PNl3FLW3jAlyPi+oaYdwd+GBG/\nbCgfRTbo4JNkSc5JxUrlNhWzJtKdx7nAz9OcMyuTzSeyCPgc2YRVbS4CjgHo4A7gFuBTqR1kBNkd\nyJ3Nrh8RM8iqyjaMiMeAW4HjyJINwPXAFyUNTPFuJGnFVP55SUNS+dqp0wHA62TJ6RBJny78YZgV\n4DsVs7cbnCY7Gkg29/evgbYpcs8mmwzpEOAvLB6/i4iYLelh4OoOznsVWVvIvWTjfX09ImYViOcf\nLE5et5DN9HdrWj+PbNiRqWnk3TnAfhHxV0nvAf6eFfMK2V3UwhTrfEl7ATek9pymM1GaFeVhWsxK\nImkFspkDR0dE5fNamFXB1V9mJZD0YeBh4GdOKNaf+U7FzMxK4zsVMzMrjZOKmZmVxknFzMxK46Ri\nZmalcVIxM7PSOKmYmVlp/j9ruWYWoc0OMQAAAABJRU5ErkJggg==\n", - "text/plain": [ - "
    " - ] - }, - "metadata": { - "tags": [] - }, - "output_type": "display_data" - } - ], - "source": [ - "cp.sort_values('Count', ascending=False).plot(kind='bar', color= 'red', x='Day_of_week', y='Count')\n", - "plt.xlabel(\"Day of week\")\n", - "plt.ylabel(\"Number of reported crimes\")\n", - "plt.title(\"No.of reported crimes per day\")" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "colab_type": "text", - "id": "x62BiCgBMOtq" - }, - "source": [ - "\n", - "# RDD" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "colab_type": "text", - "id": "VGXK6uEuUKRh" - }, - "source": [ - "> With map, you define a function and then apply it record by record. Flatmap returns a new RDD by first applying a function to all of the elements in RDDs and then flattening the result. Filter, returns a new RDD. Meaning only the elements that satisfy a condition. With reduce, we are taking neighboring elements and producing a single combined result.\n", - "For example, let's say you have a set of numbers. You can reduce this to its sum by providing a function that takes as input two values and reduces them to one. \n", - "\n", - "Some of the reasons you would use a dataframe over RDD are:\n", - "
      \n", - "
    1. It's ability to represnt data as rows and columns. But this also means it can only hold structred and semi-structured data.
      2. \n", - "
    2. It allows processing data in different formats (AVRO, CSV, JSON, and storage system HDFS, HIVE tables, MySQL).
      3. \n", - "
    3. It's superior job Optimization capability.
      4. \n", - "
    4. DataFrame API is very easy to use.
      5. " - ] - }, - { - "cell_type": "code", - "execution_count": 0, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 51 - }, - "colab_type": "code", - "id": "0_WvAgyvR7m6", - "outputId": "a89a2324-9d80-4b70-c8f7-ad1ddc8f4956" - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "DISTRICT,DISTRICT NAME,ADDRESS,CITY,STATE,ZIP,WEBSITE,PHONE,FAX,TTY,X COORDINATE,Y COORDINATE,LATITUDE,LONGITUDE,LOCATION\n", - "1,Central,1718 S State St,Chicago,IL,60616,http://home.chicagopolice.org/community/districts/1st-district-central/,312-745-4290,312-745-3694,312-745-3693,1176569.052,1891771.704,41.85837259,-87.62735617,\"(41.8583725929, -87.627356171)\"\n" - ] - } - ], - "source": [ - "psrdd = sc.textFile('police-station.csv')\n", - "print(psrdd.first())\n", - "ps_header = psrdd.first()\n", - "ps_rest = psrdd.filter(lambda line: line!=ps_header)\n", - "print(ps_rest.first())" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "colab_type": "text", - "id": "P65eAFO3Mkdd" - }, - "source": [ - "**How many police stations are there?**" - ] - }, - { - "cell_type": "code", - "execution_count": 0, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 34 - }, - "colab_type": "code", - "id": "Vi03EU0CMSmO", - "outputId": "5c566a1f-86ae-4175-956c-c6b2f6b7d692" - }, - "outputs": [ - { - "data": { - "text/plain": [ - "24" - ] - }, - "execution_count": 53, - "metadata": { - "tags": [] - }, - "output_type": "execute_result" - } - ], - "source": [ - "ps_rest.map(lambda line: line.split(\",\")).count()" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "colab_type": "text", - "id": "3c4bci70MnlQ" - }, - "source": [ - "**Display the District ID, District name, Address and Zip for the police station with District ID 7**" - ] - }, - { - "cell_type": "code", - "execution_count": 0, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 34 - }, - "colab_type": "code", - "id": "fWFpo_WxMnvm", - "outputId": "cae005e4-efb8-4bf9-c3fe-caff8a69710c" - }, - "outputs": [ - { - "data": { - "text/plain": [ - "[('7', 'Englewood', '1438 W 63rd St', '60636')]" - ] - }, - "execution_count": 54, - "metadata": { - "tags": [] - }, - "output_type": "execute_result" - } - ], - "source": [ - "# District is column 0\n", - "(ps_rest.filter(lambda line: line.split(\",\")[0]=='7').\n", - " map(lambda line: (line.split(\",\")[0],\n", - " line.split(\",\")[1],\n", - " line.split(\",\")[2],\n", - " line.split(\",\")[5])).collect())" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "colab_type": "text", - "id": "ZYmb5FscMph3" - }, - "source": [ - "**Police stations 10 and 11 are geographically close to each other. Display the District ID, District name, address and zip code**" - ] - }, - { - "cell_type": "code", - "execution_count": 0, - "metadata": { + "nbformat": 4, + "nbformat_minor": 0, + "metadata": { "colab": { - "base_uri": "https://localhost:8080/", - "height": 51 - }, - "colab_type": "code", - "id": "6ZcRIX3mMquF", - "outputId": "319981cb-4fae-4945-dc43-100b248144a5", - "scrolled": true - }, - "outputs": [ - { - "data": { - "text/plain": [ - "[('11', 'Harrison', '3151 W Harrison St', '60612'),\n", - " ('10', 'Ogden', '3315 W Ogden Ave', '60623')]" - ] - }, - "execution_count": 55, - "metadata": { - "tags": [] - }, - "output_type": "execute_result" + "name": "Colab and PySpark.ipynb", + "provenance": [], + "collapsed_sections": [ + "YR1CO3FTqO5h", + "K_q3Yzc9qYc0", + "5gs9JXCWqb9s", + "NhM3wLG2qhlN", + "1NJMWs4NqnlF", + "6Ul54hAYyHyd", + "VOqLNkRKyUIS", + "X6zdrH15_CCW", + "Dd6t0uFzuR4X", + "hmIqq6xPK7m7", + "HgwoX-pfNqQI", + "_QwZtWxZRCBn", + "eFoagdqARKb8", + "rsD48rckdHPe", + "ikGR5pDICTu7", + "85Lv3zSXCcOY", + "QlMf04i2CjDC", + "4CDifVC2Cnml", + "CbpEj9fECrW3", + "WbKK5iHwmIoV", + "9bKlvX-SH-Wy", + "zLU-a4auIEvh", + "-069UYUwIIYI", + "aN0-A_JsIX-X", + "xOQPOt19q_he", + "aHjILb1DriuX", + "PIKigra7A34e", + "ldtA0wk9BMkT", + "KQ6Ul9HGCwC3", + "sY6PstyLDp6P", + "7OZElEvcGOD1", + "EEEB2TVqL4Ie", + "HNPhsx8P2tUH", + "x62BiCgBMOtq", + "3wn2zXe7TbI3", + "6z9gkoE2R1m1", + "VZ1bYvF8R8Dc", + "oVwGYAZZiyGV", + "f3crkAQVlxKp", + "vIbXZT29JxmG", + "dABRu9eokZxw", + "xrgbAiZHnq_U", + "244El832wT8f", + "H5cFCvbczHz_" + ] + }, + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.6.5" } - ], - "source": [ - "# District is column 0\n", - "(ps_rest.filter(lambda line: line.split(\",\")[0] in ['10', '11']).\n", - " map(lambda line: (line.split(\",\")[0],\n", - " line.split(\",\")[1],\n", - " line.split(\",\")[2],\n", - " line.split(\",\")[5])).collect())" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "\n", - "# User-Defined Functions (UDF)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "PySpark User-Defined Functions (UDFs) help you convert your python code into a scalable version of itself. It comes in handy more than you can imagine, but beware, as the performance is less when you compare it with pyspark functions. You can view examples of how UDF works [here](https://docs.databricks.com/spark/latest/spark-sql/udf-python.html). What I will give in this section is some theory on how it works, and why it is slower.\n", - "\n", - "When you try to run a UDF in PySpark, each executor creates a python process. Data will be serialised and deserialised between each executor and python. This leads to lots of performance impact and overhead on spark jobs, making it less efficent than using spark dataframes. Apart from this, sometimes you might have memory issues while using UDFs. The Python worker consumes huge off-heap memory and so it often leads to memoryOverhead, thereby failing your job. Keeping these in mind, I wouldn't recommend using them, but at the end of the day, your choice." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "\n", - "# Common Questions" - ] }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Recommeded IDE\n", - "I personally prefer [PyCharm](https://www.jetbrains.com/pycharm/) while coding in Python/PySpark. It's based on IntelliJ IDEA so it has a lot of features! And the main advantage I have felt is the ease of installing pyspark and other packages. You can customize it with themes and plugins, and it lets you enhance productivity while coding by providing some features like suggestions, Local VCS etc." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Submitting a spark job\n", - "The python syntax for running jobs is: `python .py ...`
      \n", - "But when you submit a spark job you have to use `spark-submit` to run the application. \n", - "\n", - "Here is a simple example of a spark-submit command:
      \n", - "`spark-submit filename.py --named_argument 'arguemnt value'`
      \n", - "Here, named_argument is an arguemnt that you are reading from inside your script. \n", - "\n", - "There are other options you can pass in the command, like:
      \n", - "`--py-files` which helps you pass a python file to read in your file,
      \n", - "`--files` which helps pass other files like txt or config,
      \n", - "`--deploy-mode` which tells wether to deploy your worker node on cluster or locally\n", - "`--conf` which helps pass different configurations, like memoryOverhead, dynamicAllocation etc.\n", - "\n", - "\n", - "There is an [entire page](https://spark.apache.org/docs/latest/submitting-applications.html) in spark documentation dedicated to this. I highly recommend you go through it once." - ] - } - ], - "metadata": { - "colab": { - "collapsed_sections": [ - "qJoeN3e8_Gzk", - "6Ul54hAYyHyd", - "X6zdrH15_CCW", - "Dd6t0uFzuR4X", - "hmIqq6xPK7m7", - "HgwoX-pfNqQI", - "_QwZtWxZRCBn", - "eFoagdqARKb8", - "rsD48rckdHPe", - "WbKK5iHwmIoV", - "xOQPOt19q_he", - "aHjILb1DriuX", - "PIKigra7A34e", - "ldtA0wk9BMkT", - "KQ6Ul9HGCwC3", - "sY6PstyLDp6P", - "7OZElEvcGOD1", - "EEEB2TVqL4Ie", - "x62BiCgBMOtq" - ], - "name": "Colab and PySpark.ipynb", - "provenance": [] - }, - "kernelspec": { - "display_name": "Python 2", - "language": "python", - "name": "python2" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 2 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython2", - "version": "2.7.16" - } - }, - "nbformat": 4, - "nbformat_minor": 1 -} + "cells": [ + { + "cell_type": "markdown", + "metadata": { + "id": "iox_ufgbqDXa", + "colab_type": "text" + }, + "source": [ + "

      Introduction to Google Colab and PySpark

      " + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "1qV6Grv7qIa9", + "colab_type": "text" + }, + "source": [ + "## Table Of Contents:\n", + "
        \n", + "
      1. Objective
        2. \n", + "
      2. Prerequisite
        3. \n", + "
      3. Notes from the author
        4. \n", + "
      4. Big data, PySpark and Colaboratory\n", + "
          \n", + "
        1. Big data
          2. \n", + "
        2. PySpark
          3. \n", + "
        3. Colaboratory
          4. \n", + "
        \n", + "
        5. \n", + "
      5. Jupyter notebook basics\n", + "
          \n", + "
        1. Code cells
          2. \n", + "
        2. Text cells
          3. \n", + "
        3. Access to the shell
          4. \n", + "
        4. Install Spark
          5. \n", + "
        \n", + "
        6. \n", + "
      6. Loading Dataset
        7. \n", + "
      7. Working with the DataFrame API\n", + "
          \n", + "
        1. Viewing Dataframe
          2. \n", + "
        2. Schema of a DataFrame
          3. \n", + "
        3. Working with Columns\n", + " \n", + "
          4. \n", + "
        4. Working with Rows\n", + " \n", + "
          5. \n", + "
        \n", + "
        8. \n", + "
      8. Hands-on Questions
        9. \n", + "
      9. Functions\n", + "
          \n", + "
        1. String Functions
          2. \n", + "
        2. Numeric functions
          3. \n", + "
        3. Date
          4. \n", + "
        \n", + "
        10. \n", + "
      10. Working with Dates
        11. \n", + "
      11. Working with joins
        12. \n", + "
      12. Hands-on again!
        13. \n", + "
      13. Spark SQL
        14. \n", + "
      14. RDD
        15. \n", + "
      15. User-Defined Functions (UDF)
        16. \n", + "
      16. Common Questions\n", + "
          \n", + "
        1. Recommended IDE
          2. \n", + "
        2. Submitting a Spark Job
          3. \n", + "
        3. Creating Dataframes
          4. \n", + "
        4. Drop Duplicates
          5. \n", + "
        5. Fine Tuning a PySpark Job\n", + " \n", + "
          6. \n", + "
        \n", + "
        17. \n", + "
      " + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "FFnYZltvqLgt", + "colab_type": "text" + }, + "source": [ + "\n", + "## Objective\n", + "The objective of this notebook is to:\n", + ">
    5. Give a proper understanding about the different PySpark functions available.
      6. \n", + ">
    6. A short introduction to Google Colab, as that is the platform on which this notebook is written on.
      7. \n", + "\n", + "Once you complete this notebook, you should be able to write pyspark programs in an efficent way. The ideal way to use this is by going through the examples given and then trying them on Colab. At the end there are a few hands on questions which you can use to evaluate yourself." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "YR1CO3FTqO5h", + "colab_type": "text" + }, + "source": [ + "\n", + "## Prerequisite\n", + ">
    7. Although some theory about pyspark and big data will be given in this notebook, I recommend everyone to read more about it and have a deeper understanding on how the functions get executed and the relevance of big data in the current scenario.\n", + ">
    8. A good understanding on python will be an added bonus." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "bGbIBPLHqVXc", + "colab_type": "text" + }, + "source": [ + "\n", + "### Notes from the author\n", + "\n", + "This tutorial was made using Google Colab so the code you see here is meant to run on a colab notebook.
      \n", + "It goes through basic [PySpark Functions](https://spark.apache.org/docs/latest/api/python/index.html) and a short introduction on how to use [Colab](https://colab.research.google.com/notebooks/basic_features_overview.ipynb).
      \n", + "If you want to view my colab notebook for this particular tutorial, you can view it [here](https://colab.research.google.com/drive/1G894WS7ltIUTusWWmsCnF_zQhQqZCDOc). The viewing experience and readability is much better there.
      \n", + "If you want to try out things with this notebook as a base, feel free to download it from my repo [here](https://github.com/jacobceles/knowledge-repo/blob/master/pyspark/Colab%20and%20PySpark.ipynb) and then use it with jupyter notebook." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "K_q3Yzc9qYc0", + "colab_type": "text" + }, + "source": [ + "\n", + "## Big data, PySpark and Colaboratory" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "5gs9JXCWqb9s", + "colab_type": "text" + }, + "source": [ + "\n", + "### Big data" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "q3UkLT6yqebl", + "colab_type": "text" + }, + "source": [ + "Big data usually means data of such huge volume that normal data storage solutions cannot efficently store and process it. In this era, data is being generated at an absurd rate. Data is collected for each movement a person makes. The bulk of big data comes from three primary sources: \n", + "
        \n", + "
      1. Social data
        2. \n", + "
      2. Machine data
        3. \n", + "
      3. Transactional data
        4. \n", + "
      \n", + "\n", + "Some common examples for the sources of such data include internet searches, facebook posts, doorbell cams, smartwatches, online shopping history etc. Every action creates data, it is just a matter of of there is a way to collect them or not. But what's interesting is that out of all this data collected, not even 5% of it is being used fully. There is a huge demand for big data professionals in the industry. Even though the number of graduates with a specialization in big data are rising, the problem is that they don't have the practical knowledge about big data scenarios, which leads to bad architecutres and inefficent methods of processing data.\n", + "\n", + ">If you are interested to know more about the landscape and technologies involved, here is [an article](https://hostingtribunal.com/blog/big-data-stats/) which I found really interesting!" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "NhM3wLG2qhlN", + "colab_type": "text" + }, + "source": [ + "\n", + "### PySpark" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "VBfC_kvjqjPj", + "colab_type": "text" + }, + "source": [ + "If you are working in the field of big data, you must have definelty heard of spark. If you look at the [Apache Spark](https://spark.apache.org/) website, you will see that it is said to be a `Lightning-fast unified analytics engine`. PySPark is a flavour of Spark used for processing and analysing massive volumes of data. If you are familiar with python and have tried it for huge datasets, you should know that the execution time can get ridiculous. Enter PySpark!\n", + "\n", + "Imagine your data resides in a distributed manner at different places. If you try brining your data to one point and executing your code there, not only would that be inefficent, but also cause memory issues. Now let's say your code goes to the data rather than the data coming to where your code. This will help avoid unneccesary data movement which will thereby decrease the running time. \n", + "\n", + "PySpark is the Python API of Spark; which means it can do almost all the things python can. Machine learning(ML) pipelines, exploratory data analysis (at scale), ETLs for data platform, and much more! And all of them in a distributed manner. One of the best parts of pyspark is that if you are already familiar with python, it's really easy to learn.\n", + "\n", + "Apart from PySpark, there is another language called Scala used for big data processing. Scala is frequently over 10 times faster than Python is native for Hadoop as its based on JVM. But PySpark is getting adopted at a fast rate because of the ease of use, easier learning curve and ML capabilities.\n", + "\n", + "I will briefly explain how a PySpark job works, but I strongly recommend you read more about the [architecture](https://data-flair.training/blogs/how-apache-spark-works/) and how everything works. Now, before I get into it, let me talk about some basic jargons first:\n", + "\n", + "Cluster is a set of loosely or tightly connected computers that work together so that they can be viewed as a single system.\n", + "\n", + "Hadoop is an open source, scalable, and fault tolerant framework written in Java. It efficiently processes large volumes of data on a cluster of commodity hardware. Hadoop is not only a storage system but is a platform for large data storage as well as processing.\n", + "\n", + "HDFS (Hadoop distributed file system). It is one of the world's most reliable storage system. HDFS is a Filesystem of Hadoop designed for storing very large files running on a cluster of commodity hardware.\n", + "\n", + "MapReduce is a data Processing framework, which has 2 phases - Mapper and Reducer. The map procedure performs filtering and sorting, and the reduce method performs a summary operation. It usually runs on a hadoop cluster.\n", + "\n", + "Transformation refers to the operations applied on a dataset to create a new dataset. Filter, groupBy and map are the examples of transformations.\n", + "\n", + "Actions Actions refer to an operation which instructs Spark to perform computation and send the result back to driver. This is an example of action.\n", + "\n", + "Alright! Now that that's out of the way, let me explain how a spark job runs. In simple terma, each time you submit a pyspark job, the code gets internally converted into a MapReduce program and gets executed in the Java virtual machine. Now one of the thoughts that might be popping in your mind will probably be:
      `So the code gets converted into a MapReduce program. Wouldn't that mean MapReduce is faster than pySpark?`
      Well, the answer is a big NO. This is what makes spark jobs special. Spark is capable of handling a massive amount of data at a time, in it's distributed environment. It does this through in-memory processing, which is what makes it almost 100 times faster than Hadoop. Another factor which amkes it fast is Lazy Evaluation. Spark delays its evaluation as much as it can. Each time you submit a job, spark creates an action plan for how to execute the code, and then does nothing. Finally, when you ask for the result(i.e, calls an action), it executes the plan, which is basically all the transofrmations you have mentioned in your code. That's basically the gist of it. \n", + "\n", + "Now lastly, I want to talk about on more thing. Spark mainly consists of 4 modules:\n", + "\n", + "
        \n", + "
      1. Spark SQL - helps to write spark programs using SQL like queries.
        2. \n", + "
      2. Spark Streaming - is an extension of the core Spark API that enables scalable, high-throughput, fault-tolerant stream processing of live data streams. used heavily in processing of social media data.
        3. \n", + "
      3. Spark MLLib - is the machine learning component of SPark. It helps train ML models on massive datasets with very high efficeny.
        4. \n", + "
      4. Spark GraphX - is the visualization component of Spark. It enables users to view data both as graphs and as collections without data movement or duplication.
        5. \n", + "
      \n", + "\n", + "Hopefully this image gives a better idea of what I am talking about:\n", + "\"Spark\n", + "
      Source: Datanami
      \n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "1NJMWs4NqnlF", + "colab_type": "text" + }, + "source": [ + "\n", + "### Colaboratory" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "ynKFk6b7qoWr", + "colab_type": "text" + }, + "source": [ + "In the words of Google:
      \n", + "`Colaboratory, or “Colab” for short, is a product from Google Research. Colab allows anybody to write and execute arbitrary python code through the browser, and is especially well suited to machine learning, data analysis and education. More technically, Colab is a hosted Jupyter notebook service that requires no setup to use, while providing free access to computing resources including GPUs.`\n", + "\n", + "The reason why I used colab is because of its shareability and free GPU. Yeah you read that right. A FREE GPU! Additionally, it helps use different Google services conveniently. It saves to Google Drive and all the services are very closely related. I recommend you go through the offical [overview documentation](https://colab.research.google.com/notebooks/basic_features_overview.ipynb) if you want to know more about it.\n", + "If you have more questions about colab, please [refer this link](https://research.google.com/colaboratory/faq.html).\n", + "\n", + ">While using a colab notebook, you will need an active internet connection to keep a session alive. If you lose the connection you will have to download the datasets again." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "_N5-lspH_N8B" + }, + "source": [ + "\n", + "## Jupyter notebook basics" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "6Ul54hAYyHyd" + }, + "source": [ + "\n", + "### Code cells" + ] + }, + { + "cell_type": "code", + "metadata": { + "colab_type": "code", + "id": "j38beRUTCI5c", + "outputId": "18933a7a-e345-4ad9-f54f-62cef10b999e", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 34 + } + }, + "source": [ + "2*3" + ], + "execution_count": 0, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "6" + ] + }, + "metadata": { + "tags": [] + }, + "execution_count": 1 + } + ] + }, + { + "cell_type": "code", + "metadata": { + "colab_type": "code", + "id": "_Jewe_e9CIYa", + "colab": {} + }, + "source": [ + "import pandas as pd" + ], + "execution_count": 0, + "outputs": [] + }, + { + "cell_type": "code", + "metadata": { + "colab_type": "code", + "id": "g8Y7w6_CCIIT", + "outputId": "eb01c5ba-4ec1-4b6a-c988-852774adc0f8", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 34 + } + }, + "source": [ + "print(\"Hello!\")" + ], + "execution_count": 0, + "outputs": [ + { + "output_type": "stream", + "text": [ + "Hello!\n" + ], + "name": "stdout" + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "VOqLNkRKyUIS" + }, + "source": [ + "\n", + "### Text cells" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "IfbaUe-oq7DK", + "colab_type": "text" + }, + "source": [ + "Hello world!" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "X6zdrH15_CCW" + }, + "source": [ + "\n", + "## Access to the shell" + ] + }, + { + "cell_type": "code", + "metadata": { + "colab_type": "code", + "id": "zdO9sjSdEVnr", + "outputId": "0fa0cc8d-9130-4003-9858-cc6785719e22", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 34 + } + }, + "source": [ + "ls" + ], + "execution_count": 0, + "outputs": [ + { + "output_type": "stream", + "text": [ + "\u001b[0m\u001b[01;34msample_data\u001b[0m/\n" + ], + "name": "stdout" + } + ] + }, + { + "cell_type": "code", + "metadata": { + "colab_type": "code", + "id": "QF9e3lDDEX3I", + "outputId": "dae9fdc8-9495-4baf-f136-27cd3d182885", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 34 + } + }, + "source": [ + "pwd" + ], + "execution_count": 0, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "'/content'" + ] + }, + "metadata": { + "tags": [] + }, + "execution_count": 5 + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "Dd6t0uFzuR4X" + }, + "source": [ + "\n", + "## Install Spark" + ] + }, + { + "cell_type": "code", + "metadata": { + "colab_type": "code", + "id": "tt7ZS1_wGgjn", + "outputId": "6c729948-f507-40af-e29d-2cec61f03ee2", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 476 + } + }, + "source": [ + "!apt-get update\n", + "!apt-get install openjdk-8-jdk-headless -qq > /dev/null\n", + "!wget -q http://archive.apache.org/dist/spark/spark-2.4.5/spark-2.4.5-bin-hadoop2.7.tgz\n", + "!tar xf spark-2.4.5-bin-hadoop2.7.tgz\n", + "!pip install -q findspark" + ], + "execution_count": 0, + "outputs": [ + { + "output_type": "stream", + "text": [ + "\r0% [Working]\r \rIgn:1 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64 InRelease\n", + "\r0% [Connecting to archive.ubuntu.com (91.189.88.152)] [Connecting to security.u\r \rIgn:2 https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64 InRelease\n", + "\r0% [Connecting to archive.ubuntu.com (91.189.88.152)] [Waiting for headers] [Co\r \rHit:3 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64 Release\n", + "\r0% [Connecting to archive.ubuntu.com (91.189.88.152)] [Waiting for headers] [Co\r0% [Release.gpg gpgv 564 B] [Waiting for headers] [Waiting for headers] [Connec\r \rHit:4 https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64 Release\n", + "Get:5 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB]\n", + "Hit:7 http://archive.ubuntu.com/ubuntu bionic InRelease\n", + "Get:8 http://ppa.launchpad.net/graphics-drivers/ppa/ubuntu bionic InRelease [21.3 kB]\n", + "Get:10 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB]\n", + "Get:11 http://ppa.launchpad.net/marutter/c2d4u3.5/ubuntu bionic InRelease [15.4 kB]\n", + "Get:12 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages [908 kB]\n", + "Get:13 https://cloud.r-project.org/bin/linux/ubuntu bionic-cran35/ InRelease [3,626 B]\n", + "Get:14 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB]\n", + "Get:15 http://ppa.launchpad.net/graphics-drivers/ppa/ubuntu bionic/main amd64 Packages [37.4 kB]\n", + "Get:16 https://cloud.r-project.org/bin/linux/ubuntu bionic-cran35/ Packages [91.7 kB]\n", + "Get:17 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages [1,205 kB]\n", + "Get:18 http://security.ubuntu.com/ubuntu bionic-security/restricted amd64 Packages [52.4 kB]\n", + "Get:19 http://security.ubuntu.com/ubuntu bionic-security/multiverse amd64 Packages [8,505 B]\n", + "Get:20 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 Packages [844 kB]\n", + "Get:21 http://ppa.launchpad.net/marutter/c2d4u3.5/ubuntu bionic/main Sources [1,813 kB]\n", + "Get:22 http://archive.ubuntu.com/ubuntu bionic-updates/restricted amd64 Packages [66.7 kB]\n", + "Get:23 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages [1,376 kB]\n", + "Get:24 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse amd64 Packages [19.8 kB]\n", + "Get:25 http://archive.ubuntu.com/ubuntu bionic-backports/main amd64 Packages [8,286 B]\n", + "Get:26 http://archive.ubuntu.com/ubuntu bionic-backports/universe amd64 Packages [8,158 B]\n", + "Get:27 http://ppa.launchpad.net/marutter/c2d4u3.5/ubuntu bionic/main amd64 Packages [875 kB]\n", + "Fetched 7,606 kB in 8s (1,014 kB/s)\n", + "Reading package lists... Done\n" + ], + "name": "stdout" + } + ] + }, + { + "cell_type": "code", + "metadata": { + "colab_type": "code", + "id": "sdOOq4twHN1K", + "colab": {} + }, + "source": [ + "import os\n", + "os.environ[\"JAVA_HOME\"] = \"/usr/lib/jvm/java-8-openjdk-amd64\"\n", + "os.environ[\"SPARK_HOME\"] = \"/content/spark-2.4.5-bin-hadoop2.7\"" + ], + "execution_count": 0, + "outputs": [] + }, + { + "cell_type": "code", + "metadata": { + "colab_type": "code", + "id": "3ACYMwhgHTYz", + "outputId": "ff3bc469-5ee7-4204-ee8d-dc667ff56244", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 34 + } + }, + "source": [ + "!ls" + ], + "execution_count": 0, + "outputs": [ + { + "output_type": "stream", + "text": [ + "sample_data spark-2.4.5-bin-hadoop2.7\tspark-2.4.5-bin-hadoop2.7.tgz\n" + ], + "name": "stdout" + } + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "KR1zLBk1998Z", + "colab_type": "code", + "outputId": "3fd931c5-4576-4fa4-dd2e-dc4331ca58da", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 193 + } + }, + "source": [ + "import findspark\n", + "findspark.init()\n", + "from pyspark import SparkContext\n", + "\n", + "sc = SparkContext.getOrCreate()\n", + "sc" + ], + "execution_count": 0, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/html": [ + "\n", + "
      \n", + "

      SparkContext

      \n", + "\n", + "

      Spark UI

      \n", + "\n", + "
      \n", + "
      Version
      \n", + "
      v2.4.5
      \n", + "
      Master
      \n", + "
      local[*]
      \n", + "
      AppName
      \n", + "
      pyspark-shell
      \n", + "
      \n", + "
      \n", + " " + ], + "text/plain": [ + "" + ] + }, + "metadata": { + "tags": [] + }, + "execution_count": 61 + } + ] + }, + { + "cell_type": "code", + "metadata": { + "colab_type": "code", + "id": "Gs7fzvxcHfvw", + "outputId": "886ca9fc-908e-4f24-ea46-97e569394275", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 216 + } + }, + "source": [ + "import pyspark\n", + "from pyspark.sql import SparkSession\n", + "\n", + "spark = SparkSession.builder.getOrCreate() \n", + "spark" + ], + "execution_count": 0, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/html": [ + "\n", + "
      \n", + "

      SparkSession - in-memory

      \n", + " \n", + "
      \n", + "

      SparkContext

      \n", + "\n", + "

      Spark UI

      \n", + "\n", + "
      \n", + "
      Version
      \n", + "
      v2.4.5
      \n", + "
      Master
      \n", + "
      local[*]
      \n", + "
      AppName
      \n", + "
      pyspark-shell
      \n", + "
      \n", + "
      \n", + " \n", + "
      \n", + " " + ], + "text/plain": [ + "" + ] + }, + "metadata": { + "tags": [] + }, + "execution_count": 63 + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "hmIqq6xPK7m7", + "colab_type": "text" + }, + "source": [ + "\n", + "# Loading Dataset" + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "hQ3zmGACLKlN", + "colab_type": "code", + "outputId": "a7c382c6-baff-434d-f204-c05cc6ade4f1", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 204 + } + }, + "source": [ + "# Downloading and preprocessing Chicago's Reported Crime Data\n", + "!wget https://data.cityofchicago.org/api/views/w98m-zvie/rows.csv?accessType=DOWNLOAD" + ], + "execution_count": 0, + "outputs": [ + { + "output_type": "stream", + "text": [ + "--2020-05-02 15:12:16-- https://data.cityofchicago.org/api/views/w98m-zvie/rows.csv?accessType=DOWNLOAD\n", + "Resolving data.cityofchicago.org (data.cityofchicago.org)... 52.206.68.26, 52.206.140.199, 52.206.140.205\n", + "Connecting to data.cityofchicago.org (data.cityofchicago.org)|52.206.68.26|:443... connected.\n", + "HTTP request sent, awaiting response... 200 OK\n", + "Length: unspecified [text/csv]\n", + "Saving to: ‘rows.csv?accessType=DOWNLOAD’\n", + "\n", + "rows.csv?accessType [ <=> ] 58.90M 3.21MB/s in 19s \n", + "\n", + "2020-05-02 15:12:41 (3.14 MB/s) - ‘rows.csv?accessType=DOWNLOAD’ saved [61759120]\n", + "\n" + ], + "name": "stdout" + } + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "Wpq2jYvIMOJy", + "colab_type": "code", + "outputId": "74d00380-469b-46da-86be-434b64e6de3e", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 51 + } + }, + "source": [ + "!ls" + ], + "execution_count": 0, + "outputs": [ + { + "output_type": "stream", + "text": [ + "'rows.csv?accessType=DOWNLOAD'\t spark-2.4.5-bin-hadoop2.7\n", + " sample_data\t\t\t spark-2.4.5-bin-hadoop2.7.tgz\n" + ], + "name": "stdout" + } + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "uCJX5cwdMS9q", + "colab_type": "code", + "colab": {} + }, + "source": [ + "#Renaming the downloaded file\n", + "!mv rows.csv?accessType=DOWNLOAD reported-crimes.csv" + ], + "execution_count": 0, + "outputs": [] + }, + { + "cell_type": "code", + "metadata": { + "id": "hz6ALr5mMqZt", + "colab_type": "code", + "outputId": "ce1ae7ef-8a2c-431c-e88a-329ac9060bbd", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 224 + } + }, + "source": [ + "df = spark.read.csv('reported-crimes.csv',header=True)\n", + "spark.conf.set(\"spark.sql.repl.eagerEval.enabled\", True)\n", + "df.show(5)" + ], + "execution_count": 0, + "outputs": [ + { + "output_type": "stream", + "text": [ + "+--------+-----------+--------------------+--------------------+----+------------------+--------------------+--------------------+------+--------+----+--------+----+--------------+--------+------------+------------+----+--------------------+------------+-------------+--------------------+\n", + "| ID|Case Number| Date| Block|IUCR| Primary Type| Description|Location Description|Arrest|Domestic|Beat|District|Ward|Community Area|FBI Code|X Coordinate|Y Coordinate|Year| Updated On| Latitude| Longitude| Location|\n", + "+--------+-----------+--------------------+--------------------+----+------------------+--------------------+--------------------+------+--------+----+--------+----+--------------+--------+------------+------------+----+--------------------+------------+-------------+--------------------+\n", + "|12040452| JD220630|11/21/2019 12:00:...| 004XX N WABASH AVE|1153|DECEPTIVE PRACTICE|FINANCIAL IDENTIT...| RESIDENCE| false| false|1834| 018| 42| 8| 11| null| null|2019|05/01/2020 03:49:...| null| null| null|\n", + "|12039199| JD219081|04/01/2019 08:00:...| 021XX S INDIANA AVE|1153|DECEPTIVE PRACTICE|FINANCIAL IDENTIT...| RESIDENCE| false| false|0132| 001| 3| 33| 11| 1177977| 1890072|2019|05/01/2020 03:48:...|41.853676611|-87.622239838|(41.853676611, -8...|\n", + "|12040466| JD220650|11/07/2019 12:00:...|026XX N NEW ENGLA...|1152|DECEPTIVE PRACTICE|ILLEGAL USE CASH ...| RESIDENCE| false| false|2512| 025| 36| 18| 11| null| null|2019|05/01/2020 03:49:...| null| null| null|\n", + "|11938499| JD100565|12/31/2019 11:30:...|070XX S COTTAGE G...|0460| BATTERY| SIMPLE| OTHER (SPECIFY)| true| false|0321| 003| 6| 42| 08B| 1182773| 1858454|2019|05/01/2020 03:48:...| 41.76680404|-87.605619683|(41.76680404, -87...|\n", + "|11864640| JC477069|10/18/2019 09:56:...|002XX S LAVERGNE AVE|041A| BATTERY|AGGRAVATED - HANDGUN| SIDEWALK| false| false|1533| 015| 28| 25| 04B| 1143293| 1898684|2019|05/01/2020 03:48:...|41.878026693|-87.749328671|(41.878026693, -8...|\n", + "+--------+-----------+--------------------+--------------------+----+------------------+--------------------+--------------------+------+--------+----+--------+----+--------------+--------+------------+------------+----+--------------------+------------+-------------+--------------------+\n", + "only showing top 5 rows\n", + "\n" + ], + "name": "stdout" + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "HgwoX-pfNqQI", + "colab_type": "text" + }, + "source": [ + "\n", + "# Working with the Dataframe API" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "_QwZtWxZRCBn", + "colab_type": "text" + }, + "source": [ + "\n", + "## Viewing Dataframe" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "50LZ3S8_PMg_", + "colab_type": "text" + }, + "source": [ + "In Spark, you have a couple of options to view the DataFrame(DF).\n", + "\n", + "\n", + "1. `take(3)` will return a list of three row objects. \n", + "2. `df.collect()` will get all of the data from the entire DataFrame . Be careful when using it, because if you have a large data set when you run collect, you can easily crash the driver node. \n", + "3. If you want Spark to print out your DataFrame in a nice format, then try `df.show()` with the number of rows as paramter. You might notice that the show functions truncates the data as inthe example above. You can use the parameter `truncate=False` to show the entire data. For example, you can use `df.show(5, False)` or `df.show(truncate=False)` to disable truncation of data.\n", + "\n", + "> The limit function **returns a new DataFrame** by taking the first n rows.\n", + "\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "eFoagdqARKb8", + "colab_type": "text" + }, + "source": [ + "\n", + "## Schema of a DataFrame" + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "w6qwTjGsNxrw", + "colab_type": "code", + "outputId": "66270202-5824-4a6b-c028-528af2026bff", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 391 + } + }, + "source": [ + "df.dtypes" + ], + "execution_count": 0, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "[('ID', 'string'),\n", + " ('Case Number', 'string'),\n", + " ('Date', 'string'),\n", + " ('Block', 'string'),\n", + " ('IUCR', 'string'),\n", + " ('Primary Type', 'string'),\n", + " ('Description', 'string'),\n", + " ('Location Description', 'string'),\n", + " ('Arrest', 'string'),\n", + " ('Domestic', 'string'),\n", + " ('Beat', 'string'),\n", + " ('District', 'string'),\n", + " ('Ward', 'string'),\n", + " ('Community Area', 'string'),\n", + " ('FBI Code', 'string'),\n", + " ('X Coordinate', 'string'),\n", + " ('Y Coordinate', 'string'),\n", + " ('Year', 'string'),\n", + " ('Updated On', 'string'),\n", + " ('Latitude', 'string'),\n", + " ('Longitude', 'string'),\n", + " ('Location', 'string')]" + ] + }, + "metadata": { + "tags": [] + }, + "execution_count": 11 + } + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "CCGTFlCWRPw4", + "colab_type": "code", + "outputId": "009696e4-0f08-4630-b9d6-d8561d5d95bc", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 425 + } + }, + "source": [ + "df.printSchema()" + ], + "execution_count": 0, + "outputs": [ + { + "output_type": "stream", + "text": [ + "root\n", + " |-- ID: string (nullable = true)\n", + " |-- Case Number: string (nullable = true)\n", + " |-- Date: string (nullable = true)\n", + " |-- Block: string (nullable = true)\n", + " |-- IUCR: string (nullable = true)\n", + " |-- Primary Type: string (nullable = true)\n", + " |-- Description: string (nullable = true)\n", + " |-- Location Description: string (nullable = true)\n", + " |-- Arrest: string (nullable = true)\n", + " |-- Domestic: string (nullable = true)\n", + " |-- Beat: string (nullable = true)\n", + " |-- District: string (nullable = true)\n", + " |-- Ward: string (nullable = true)\n", + " |-- Community Area: string (nullable = true)\n", + " |-- FBI Code: string (nullable = true)\n", + " |-- X Coordinate: string (nullable = true)\n", + " |-- Y Coordinate: string (nullable = true)\n", + " |-- Year: string (nullable = true)\n", + " |-- Updated On: string (nullable = true)\n", + " |-- Latitude: string (nullable = true)\n", + " |-- Longitude: string (nullable = true)\n", + " |-- Location: string (nullable = true)\n", + "\n" + ], + "name": "stdout" + } + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "xpsaQ4JMRUiS", + "colab_type": "code", + "outputId": "72504489-13b6-4e02-91d4-8dd8d2826954", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 391 + } + }, + "source": [ + "# Defining a schema\n", + "from pyspark.sql.types import StructType, StructField, StringType, TimestampType, BooleanType, DoubleType, IntegerType\n", + "df.columns" + ], + "execution_count": 0, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "['ID',\n", + " 'Case Number',\n", + " 'Date',\n", + " 'Block',\n", + " 'IUCR',\n", + " 'Primary Type',\n", + " 'Description',\n", + " 'Location Description',\n", + " 'Arrest',\n", + " 'Domestic',\n", + " 'Beat',\n", + " 'District',\n", + " 'Ward',\n", + " 'Community Area',\n", + " 'FBI Code',\n", + " 'X Coordinate',\n", + " 'Y Coordinate',\n", + " 'Year',\n", + " 'Updated On',\n", + " 'Latitude',\n", + " 'Longitude',\n", + " 'Location']" + ] + }, + "metadata": { + "tags": [] + }, + "execution_count": 13 + } + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "ik62VX34SlFh", + "colab_type": "code", + "colab": {} + }, + "source": [ + "labels = [\n", + " ('ID',StringType()),\n", + " ('Case Number',StringType()),\n", + " ('Date',TimestampType()),\n", + " ('Block',StringType()),\n", + " ('IUCR',StringType()),\n", + " ('Primary Type',StringType()),\n", + " ('Description',StringType()),\n", + " ('Location Description',StringType()),\n", + " ('Arrest',StringType()),\n", + " ('Domestic',BooleanType()),\n", + " ('Beat',StringType()),\n", + " ('District',StringType()),\n", + " ('Ward',StringType()),\n", + " ('Community Area',StringType()),\n", + " ('FBI Code',StringType()),\n", + " ('X Coordinate',StringType()),\n", + " ('Y Coordinate',StringType()),\n", + " ('Year',IntegerType()),\n", + " ('Updated On',StringType()),\n", + " ('Latitude',DoubleType()),\n", + " ('Longitude',DoubleType()),\n", + " ('Location',StringType()),\n", + " ('Historical Wards 2003-2015',StringType()),\n", + " ('Zip Codes',StringType()),\n", + " ('Community Areas',StringType()),\n", + " ('Census Tracts',StringType()),\n", + " ('Wards',StringType()),\n", + " ('Boundaries - ZIP Codes',StringType()),\n", + " ('Police Districts',StringType()),\n", + " ('Police Beats',StringType())\n", + "]" + ], + "execution_count": 0, + "outputs": [] + }, + { + "cell_type": "code", + "metadata": { + "id": "T-Fp5y_oU9SF", + "colab_type": "code", + "outputId": "25f313fe-7ed2-416e-ea63-8c2a7deca4fc", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 54 + } + }, + "source": [ + "schema = StructType([StructField (x[0], x[1], True) for x in labels])\n", + "schema" + ], + "execution_count": 0, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "StructType(List(StructField(ID,StringType,true),StructField(Case Number,StringType,true),StructField(Date,TimestampType,true),StructField(Block,StringType,true),StructField(IUCR,StringType,true),StructField(Primary Type,StringType,true),StructField(Description,StringType,true),StructField(Location Description,StringType,true),StructField(Arrest,StringType,true),StructField(Domestic,BooleanType,true),StructField(Beat,StringType,true),StructField(District,StringType,true),StructField(Ward,StringType,true),StructField(Community Area,StringType,true),StructField(FBI Code,StringType,true),StructField(X Coordinate,StringType,true),StructField(Y Coordinate,StringType,true),StructField(Year,IntegerType,true),StructField(Updated On,StringType,true),StructField(Latitude,DoubleType,true),StructField(Longitude,DoubleType,true),StructField(Location,StringType,true),StructField(Historical Wards 2003-2015,StringType,true),StructField(Zip Codes,StringType,true),StructField(Community Areas,StringType,true),StructField(Census Tracts,StringType,true),StructField(Wards,StringType,true),StructField(Boundaries - ZIP Codes,StringType,true),StructField(Police Districts,StringType,true),StructField(Police Beats,StringType,true)))" + ] + }, + "metadata": { + "tags": [] + }, + "execution_count": 15 + } + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "sgC7gtL5VTls", + "colab_type": "code", + "outputId": "dcf5a8ba-0dc8-41c1-fd86-0fb42e272277", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 561 + } + }, + "source": [ + "df = spark.read.csv('reported-crimes.csv',schema=schema)\n", + "df.printSchema()\n", + "# The schema comes as we gave!" + ], + "execution_count": 0, + "outputs": [ + { + "output_type": "stream", + "text": [ + "root\n", + " |-- ID: string (nullable = true)\n", + " |-- Case Number: string (nullable = true)\n", + " |-- Date: timestamp (nullable = true)\n", + " |-- Block: string (nullable = true)\n", + " |-- IUCR: string (nullable = true)\n", + " |-- Primary Type: string (nullable = true)\n", + " |-- Description: string (nullable = true)\n", + " |-- Location Description: string (nullable = true)\n", + " |-- Arrest: string (nullable = true)\n", + " |-- Domestic: boolean (nullable = true)\n", + " |-- Beat: string (nullable = true)\n", + " |-- District: string (nullable = true)\n", + " |-- Ward: string (nullable = true)\n", + " |-- Community Area: string (nullable = true)\n", + " |-- FBI Code: string (nullable = true)\n", + " |-- X Coordinate: string (nullable = true)\n", + " |-- Y Coordinate: string (nullable = true)\n", + " |-- Year: integer (nullable = true)\n", + " |-- Updated On: string (nullable = true)\n", + " |-- Latitude: double (nullable = true)\n", + " |-- Longitude: double (nullable = true)\n", + " |-- Location: string (nullable = true)\n", + " |-- Historical Wards 2003-2015: string (nullable = true)\n", + " |-- Zip Codes: string (nullable = true)\n", + " |-- Community Areas: string (nullable = true)\n", + " |-- Census Tracts: string (nullable = true)\n", + " |-- Wards: string (nullable = true)\n", + " |-- Boundaries - ZIP Codes: string (nullable = true)\n", + " |-- Police Districts: string (nullable = true)\n", + " |-- Police Beats: string (nullable = true)\n", + "\n" + ], + "name": "stdout" + } + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "Dn2EAhesVmx0", + "colab_type": "code", + "outputId": "2628d775-1ddb-4c16-b684-885b84e20840", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 479 + } + }, + "source": [ + "df.show()\n", + "# This comes as null which means the datatypes we gave were wrong." + ], + "execution_count": 0, + "outputs": [ + { + "output_type": "stream", + "text": [ + "+----+-----------+----+-----+----+------------+-----------+--------------------+------+--------+----+--------+----+--------------+--------+------------+------------+----+----------+--------+---------+--------+--------------------------+---------+---------------+-------------+-----+----------------------+----------------+------------+\n", + "| ID|Case Number|Date|Block|IUCR|Primary Type|Description|Location Description|Arrest|Domestic|Beat|District|Ward|Community Area|FBI Code|X Coordinate|Y Coordinate|Year|Updated On|Latitude|Longitude|Location|Historical Wards 2003-2015|Zip Codes|Community Areas|Census Tracts|Wards|Boundaries - ZIP Codes|Police Districts|Police Beats|\n", + "+----+-----------+----+-----+----+------------+-----------+--------------------+------+--------+----+--------+----+--------------+--------+------------+------------+----+----------+--------+---------+--------+--------------------------+---------+---------------+-------------+-----+----------------------+----------------+------------+\n", + "|null| null|null| null|null| null| null| null| null| null|null| null|null| null| null| null| null|null| null| null| null| null| null| null| null| null| null| null| null| null|\n", + "|null| null|null| null|null| null| null| null| null| null|null| null|null| null| null| null| null|null| null| null| null| null| null| null| null| null| null| null| null| null|\n", + "|null| null|null| null|null| null| null| null| null| null|null| null|null| null| null| null| null|null| null| null| null| null| null| null| null| null| null| null| null| null|\n", + "|null| null|null| null|null| null| null| null| null| null|null| null|null| null| null| null| null|null| null| null| null| null| null| null| null| null| null| null| null| null|\n", + "|null| null|null| null|null| null| null| null| null| null|null| null|null| null| null| null| null|null| null| null| null| null| null| null| null| null| null| null| null| null|\n", + "|null| null|null| null|null| null| null| null| null| null|null| null|null| null| null| null| null|null| null| null| null| null| null| null| null| null| null| null| null| null|\n", + "|null| null|null| null|null| null| null| null| null| null|null| null|null| null| null| null| null|null| null| null| null| null| null| null| null| null| null| null| null| null|\n", + "|null| null|null| null|null| null| null| null| null| null|null| null|null| null| null| null| null|null| null| null| null| null| null| null| null| null| null| null| null| null|\n", + "|null| null|null| null|null| null| null| null| null| null|null| null|null| null| null| null| null|null| null| null| null| null| null| null| null| null| null| null| null| null|\n", + "|null| null|null| null|null| null| null| null| null| null|null| null|null| null| null| null| null|null| null| null| null| null| null| null| null| null| null| null| null| null|\n", + "|null| null|null| null|null| null| null| null| null| null|null| null|null| null| null| null| null|null| null| null| null| null| null| null| null| null| null| null| null| null|\n", + "|null| null|null| null|null| null| null| null| null| null|null| null|null| null| null| null| null|null| null| null| null| null| null| null| null| null| null| null| null| null|\n", + "|null| null|null| null|null| null| null| null| null| null|null| null|null| null| null| null| null|null| null| null| null| null| null| null| null| null| null| null| null| null|\n", + "|null| null|null| null|null| null| null| null| null| null|null| null|null| null| null| null| null|null| null| null| null| null| null| null| null| null| null| null| null| null|\n", + "|null| null|null| null|null| null| null| null| null| null|null| null|null| null| null| null| null|null| null| null| null| null| null| null| null| null| null| null| null| null|\n", + "|null| null|null| null|null| null| null| null| null| null|null| null|null| null| null| null| null|null| null| null| null| null| null| null| null| null| null| null| null| null|\n", + "|null| null|null| null|null| null| null| null| null| null|null| null|null| null| null| null| null|null| null| null| null| null| null| null| null| null| null| null| null| null|\n", + "|null| null|null| null|null| null| null| null| null| null|null| null|null| null| null| null| null|null| null| null| null| null| null| null| null| null| null| null| null| null|\n", + "|null| null|null| null|null| null| null| null| null| null|null| null|null| null| null| null| null|null| null| null| null| null| null| null| null| null| null| null| null| null|\n", + "|null| null|null| null|null| null| null| null| null| null|null| null|null| null| null| null| null|null| null| null| null| null| null| null| null| null| null| null| null| null|\n", + "+----+-----------+----+-----+----+------------+-----------+--------------------+------+--------+----+--------+----+--------------+--------+------------+------------+----+----------+--------+---------+--------+--------------------------+---------+---------------+-------------+-----+----------------------+----------------+------------+\n", + "only showing top 20 rows\n", + "\n" + ], + "name": "stdout" + } + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "BrkVrQUsWFzk", + "colab_type": "code", + "outputId": "8f3e831a-5c33-4cc4-bc7f-e82f59b3835c", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 224 + } + }, + "source": [ + "# So let's just stick with infered schema for now, casting date to date type on the way\n", + "from pyspark.sql.functions import col, to_timestamp\n", + "df = spark.read.csv('reported-crimes.csv',header=True).withColumn('Date',to_timestamp(col('Date'),'MM/dd/yyyy hh:mm:ss a'))\n", + "df.show(5, False)" + ], + "execution_count": 0, + "outputs": [ + { + "output_type": "stream", + "text": [ + "+--------+-----------+-------------------+-------------------------+----+------------------+-----------------------------------+--------------------+------+--------+----+--------+----+--------------+--------+------------+------------+----+----------------------+------------+-------------+-----------------------------+\n", + "|ID |Case Number|Date |Block |IUCR|Primary Type |Description |Location Description|Arrest|Domestic|Beat|District|Ward|Community Area|FBI Code|X Coordinate|Y Coordinate|Year|Updated On |Latitude |Longitude |Location |\n", + "+--------+-----------+-------------------+-------------------------+----+------------------+-----------------------------------+--------------------+------+--------+----+--------+----+--------------+--------+------------+------------+----+----------------------+------------+-------------+-----------------------------+\n", + "|12040452|JD220630 |2019-11-21 12:00:00|004XX N WABASH AVE |1153|DECEPTIVE PRACTICE|FINANCIAL IDENTITY THEFT OVER $ 300|RESIDENCE |false |false |1834|018 |42 |8 |11 |null |null |2019|05/01/2020 03:49:55 PM|null |null |null |\n", + "|12039199|JD219081 |2019-04-01 08:00:00|021XX S INDIANA AVE |1153|DECEPTIVE PRACTICE|FINANCIAL IDENTITY THEFT OVER $ 300|RESIDENCE |false |false |0132|001 |3 |33 |11 |1177977 |1890072 |2019|05/01/2020 03:48:05 PM|41.853676611|-87.622239838|(41.853676611, -87.622239838)|\n", + "|12040466|JD220650 |2019-11-07 12:00:00|026XX N NEW ENGLAND AVE |1152|DECEPTIVE PRACTICE|ILLEGAL USE CASH CARD |RESIDENCE |false |false |2512|025 |36 |18 |11 |null |null |2019|05/01/2020 03:49:55 PM|null |null |null |\n", + "|11938499|JD100565 |2019-12-31 11:30:00|070XX S COTTAGE GROVE AVE|0460|BATTERY |SIMPLE |OTHER (SPECIFY) |true |false |0321|003 |6 |42 |08B |1182773 |1858454 |2019|05/01/2020 03:48:05 PM|41.76680404 |-87.605619683|(41.76680404, -87.605619683) |\n", + "|11864640|JC477069 |2019-10-18 09:56:00|002XX S LAVERGNE AVE |041A|BATTERY |AGGRAVATED - HANDGUN |SIDEWALK |false |false |1533|015 |28 |25 |04B |1143293 |1898684 |2019|05/01/2020 03:48:05 PM|41.878026693|-87.749328671|(41.878026693, -87.749328671)|\n", + "+--------+-----------+-------------------+-------------------------+----+------------------+-----------------------------------+--------------------+------+--------+----+--------+----+--------------+--------+------------+------------+----+----------------------+------------+-------------+-----------------------------+\n", + "only showing top 5 rows\n", + "\n" + ], + "name": "stdout" + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "rsD48rckdHPe", + "colab_type": "text" + }, + "source": [ + "\n", + "# Working with Columns" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "ikGR5pDICTu7", + "colab_type": "text" + }, + "source": [ + "\n", + "#### Selecting Columns" + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "ge9-_ygideWk", + "colab_type": "code", + "outputId": "757652b2-7344-4a5b-a7ff-8cb367d13cbc", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 34 + } + }, + "source": [ + "df.Block" + ], + "execution_count": 0, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "Column" + ] + }, + "metadata": { + "tags": [] + }, + "execution_count": 19 + } + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "md5zaET8dsr4", + "colab_type": "code", + "outputId": "7d27dba2-a2ae-44e3-a38c-d2072b4a7cbd", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 34 + } + }, + "source": [ + "df['Block']" + ], + "execution_count": 0, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "Column" + ] + }, + "metadata": { + "tags": [] + }, + "execution_count": 79 + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "YxP1su8veNde", + "colab_type": "text" + }, + "source": [ + "**NOTE:**\n", + "\n", + "> **We can't always use the dot notation because this will break when the column names have reserved names or attributes to the data frame class.**\n", + "\n" + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "6Gkf14sHec9a", + "colab_type": "code", + "outputId": "6aecb1a5-e85c-4322-c2da-1852397c5c6f", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 459 + } + }, + "source": [ + "df.select(col('Block')).show(truncate=False)" + ], + "execution_count": 0, + "outputs": [ + { + "output_type": "stream", + "text": [ + "+-------------------------+\n", + "|Block |\n", + "+-------------------------+\n", + "|004XX N WABASH AVE |\n", + "|021XX S INDIANA AVE |\n", + "|026XX N NEW ENGLAND AVE |\n", + "|070XX S COTTAGE GROVE AVE|\n", + "|002XX S LAVERGNE AVE |\n", + "|024XX W FOSTER AVE |\n", + "|048XX W AUGUSTA BLVD |\n", + "|062XX S EMERALD DR |\n", + "|062XX S FRANCISCO AVE |\n", + "|120XX S LAFLIN ST |\n", + "|049XX N ALBANY AVE |\n", + "|021XX S HARDING AVE |\n", + "|026XX W BALMORAL AVE |\n", + "|071XX S CYRIL AVE |\n", + "|028XX S CHRISTIANA AVE |\n", + "|052XX S ARCHER AVE |\n", + "|057XX W ROOSEVELT RD |\n", + "|003XX S SPRINGFIELD AVE |\n", + "|002XX N MENARD AVE |\n", + "|003XX N MENARD AVE |\n", + "+-------------------------+\n", + "only showing top 20 rows\n", + "\n" + ], + "name": "stdout" + } + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "05YQ7WXhiFcm", + "colab_type": "code", + "outputId": "2df46414-e44a-45a5-b625-82f24828c9ab", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 459 + } + }, + "source": [ + "df.select(df.Block).show(truncate=False)" + ], + "execution_count": 0, + "outputs": [ + { + "output_type": "stream", + "text": [ + "+-------------------------+\n", + "|Block |\n", + "+-------------------------+\n", + "|004XX N WABASH AVE |\n", + "|021XX S INDIANA AVE |\n", + "|026XX N NEW ENGLAND AVE |\n", + "|070XX S COTTAGE GROVE AVE|\n", + "|002XX S LAVERGNE AVE |\n", + "|024XX W FOSTER AVE |\n", + "|048XX W AUGUSTA BLVD |\n", + "|062XX S EMERALD DR |\n", + "|062XX S FRANCISCO AVE |\n", + "|120XX S LAFLIN ST |\n", + "|049XX N ALBANY AVE |\n", + "|021XX S HARDING AVE |\n", + "|026XX W BALMORAL AVE |\n", + "|071XX S CYRIL AVE |\n", + "|028XX S CHRISTIANA AVE |\n", + "|052XX S ARCHER AVE |\n", + "|057XX W ROOSEVELT RD |\n", + "|003XX S SPRINGFIELD AVE |\n", + "|002XX N MENARD AVE |\n", + "|003XX N MENARD AVE |\n", + "+-------------------------+\n", + "only showing top 20 rows\n", + "\n" + ], + "name": "stdout" + } + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "NNuJ3RIqe8yY", + "colab_type": "code", + "outputId": "dbbda528-1898-44eb-b3da-744b9b0d7021", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 459 + } + }, + "source": [ + "df.select('Block','Description').show(truncate=False)" + ], + "execution_count": 0, + "outputs": [ + { + "output_type": "stream", + "text": [ + "+-------------------------+-------------------------------------------------+\n", + "|Block |Description |\n", + "+-------------------------+-------------------------------------------------+\n", + "|004XX N WABASH AVE |FINANCIAL IDENTITY THEFT OVER $ 300 |\n", + "|021XX S INDIANA AVE |FINANCIAL IDENTITY THEFT OVER $ 300 |\n", + "|026XX N NEW ENGLAND AVE |ILLEGAL USE CASH CARD |\n", + "|070XX S COTTAGE GROVE AVE|SIMPLE |\n", + "|002XX S LAVERGNE AVE |AGGRAVATED - HANDGUN |\n", + "|024XX W FOSTER AVE |FORCIBLE ENTRY |\n", + "|048XX W AUGUSTA BLVD |AGGRAVATED - HANDGUN |\n", + "|062XX S EMERALD DR |CHILD ABUSE |\n", + "|062XX S FRANCISCO AVE |AGGRAVATED - HANDGUN |\n", + "|120XX S LAFLIN ST |HARASSMENT BY ELECTRONIC MEANS |\n", + "|049XX N ALBANY AVE |FROM BUILDING |\n", + "|021XX S HARDING AVE |AGGRAVATED CRIMINAL SEXUAL ABUSE BY FAMILY MEMBER|\n", + "|026XX W BALMORAL AVE |FINANCIAL IDENTITY THEFT OVER $ 300 |\n", + "|071XX S CYRIL AVE |FROM BUILDING |\n", + "|028XX S CHRISTIANA AVE |OTHER VEHICLE OFFENSE |\n", + "|052XX S ARCHER AVE |AUTOMOBILE |\n", + "|057XX W ROOSEVELT RD |AGGRAVATED - OTHER |\n", + "|003XX S SPRINGFIELD AVE |FINANCIAL IDENTITY THEFT $300 AND UNDER |\n", + "|002XX N MENARD AVE |FINANCIAL IDENTITY THEFT $300 AND UNDER |\n", + "|003XX N MENARD AVE |FINANCIAL IDENTITY THEFT OVER $ 300 |\n", + "+-------------------------+-------------------------------------------------+\n", + "only showing top 20 rows\n", + "\n" + ], + "name": "stdout" + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "85Lv3zSXCcOY", + "colab_type": "text" + }, + "source": [ + "\n", + "#### Adding a New Column" + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "oFHUmRKZeCEV", + "colab_type": "code", + "outputId": "458c639b-f091-4fed-ce0a-466fe15a98c8", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 224 + } + }, + "source": [ + "#Adding a column in PySpark\n", + "# We are adding a column called 'One' at the end\n", + "from pyspark.sql.functions import lit\n", + "df = df.withColumn('One',lit(1))\n", + "df.show(5,truncate=False)" + ], + "execution_count": 0, + "outputs": [ + { + "output_type": "stream", + "text": [ + "+--------+-----------+-------------------+-------------------------+----+------------------+-----------------------------------+--------------------+------+--------+----+--------+----+--------------+--------+------------+------------+----+----------------------+------------+-------------+-----------------------------+---+\n", + "|ID |Case Number|Date |Block |IUCR|Primary Type |Description |Location Description|Arrest|Domestic|Beat|District|Ward|Community Area|FBI Code|X Coordinate|Y Coordinate|Year|Updated On |Latitude |Longitude |Location |One|\n", + "+--------+-----------+-------------------+-------------------------+----+------------------+-----------------------------------+--------------------+------+--------+----+--------+----+--------------+--------+------------+------------+----+----------------------+------------+-------------+-----------------------------+---+\n", + "|12040452|JD220630 |2019-11-21 12:00:00|004XX N WABASH AVE |1153|DECEPTIVE PRACTICE|FINANCIAL IDENTITY THEFT OVER $ 300|RESIDENCE |false |false |1834|018 |42 |8 |11 |null |null |2019|05/01/2020 03:49:55 PM|null |null |null |1 |\n", + "|12039199|JD219081 |2019-04-01 08:00:00|021XX S INDIANA AVE |1153|DECEPTIVE PRACTICE|FINANCIAL IDENTITY THEFT OVER $ 300|RESIDENCE |false |false |0132|001 |3 |33 |11 |1177977 |1890072 |2019|05/01/2020 03:48:05 PM|41.853676611|-87.622239838|(41.853676611, -87.622239838)|1 |\n", + "|12040466|JD220650 |2019-11-07 12:00:00|026XX N NEW ENGLAND AVE |1152|DECEPTIVE PRACTICE|ILLEGAL USE CASH CARD |RESIDENCE |false |false |2512|025 |36 |18 |11 |null |null |2019|05/01/2020 03:49:55 PM|null |null |null |1 |\n", + "|11938499|JD100565 |2019-12-31 11:30:00|070XX S COTTAGE GROVE AVE|0460|BATTERY |SIMPLE |OTHER (SPECIFY) |true |false |0321|003 |6 |42 |08B |1182773 |1858454 |2019|05/01/2020 03:48:05 PM|41.76680404 |-87.605619683|(41.76680404, -87.605619683) |1 |\n", + "|11864640|JC477069 |2019-10-18 09:56:00|002XX S LAVERGNE AVE |041A|BATTERY |AGGRAVATED - HANDGUN |SIDEWALK |false |false |1533|015 |28 |25 |04B |1143293 |1898684 |2019|05/01/2020 03:48:05 PM|41.878026693|-87.749328671|(41.878026693, -87.749328671)|1 |\n", + "+--------+-----------+-------------------+-------------------------+----+------------------+-----------------------------------+--------------------+------+--------+----+--------+----+--------------+--------+------------+------------+----+----------------------+------------+-------------+-----------------------------+---+\n", + "only showing top 5 rows\n", + "\n" + ], + "name": "stdout" + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "QlMf04i2CjDC", + "colab_type": "text" + }, + "source": [ + "\n", + "#### Renaming a Column" + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "QJqgy6lKfk2o", + "colab_type": "code", + "outputId": "a6b547cb-c643-4620-bb14-902a465bcb49", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 479 + } + }, + "source": [ + "#Renaming a column in PySpark\n", + "df = df.withColumnRenamed('One', 'Test')\n", + "df.show(truncate=False)" + ], + "execution_count": 0, + "outputs": [ + { + "output_type": "stream", + "text": [ + "+--------+-----------+-------------------+-------------------------+----+--------------------------+-------------------------------------------------+--------------------------------------+------+--------+----+--------+----+--------------+--------+------------+------------+----+----------------------+------------+-------------+-----------------------------+----+\n", + "|ID |Case Number|Date |Block |IUCR|Primary Type |Description |Location Description |Arrest|Domestic|Beat|District|Ward|Community Area|FBI Code|X Coordinate|Y Coordinate|Year|Updated On |Latitude |Longitude |Location |Test|\n", + "+--------+-----------+-------------------+-------------------------+----+--------------------------+-------------------------------------------------+--------------------------------------+------+--------+----+--------+----+--------------+--------+------------+------------+----+----------------------+------------+-------------+-----------------------------+----+\n", + "|12040452|JD220630 |2019-11-21 12:00:00|004XX N WABASH AVE |1153|DECEPTIVE PRACTICE |FINANCIAL IDENTITY THEFT OVER $ 300 |RESIDENCE |false |false |1834|018 |42 |8 |11 |null |null |2019|05/01/2020 03:49:55 PM|null |null |null |1 |\n", + "|12039199|JD219081 |2019-04-01 08:00:00|021XX S INDIANA AVE |1153|DECEPTIVE PRACTICE |FINANCIAL IDENTITY THEFT OVER $ 300 |RESIDENCE |false |false |0132|001 |3 |33 |11 |1177977 |1890072 |2019|05/01/2020 03:48:05 PM|41.853676611|-87.622239838|(41.853676611, -87.622239838)|1 |\n", + "|12040466|JD220650 |2019-11-07 12:00:00|026XX N NEW ENGLAND AVE |1152|DECEPTIVE PRACTICE |ILLEGAL USE CASH CARD |RESIDENCE |false |false |2512|025 |36 |18 |11 |null |null |2019|05/01/2020 03:49:55 PM|null |null |null |1 |\n", + "|11938499|JD100565 |2019-12-31 11:30:00|070XX S COTTAGE GROVE AVE|0460|BATTERY |SIMPLE |OTHER (SPECIFY) |true |false |0321|003 |6 |42 |08B |1182773 |1858454 |2019|05/01/2020 03:48:05 PM|41.76680404 |-87.605619683|(41.76680404, -87.605619683) |1 |\n", + "|11864640|JC477069 |2019-10-18 09:56:00|002XX S LAVERGNE AVE |041A|BATTERY |AGGRAVATED - HANDGUN |SIDEWALK |false |false |1533|015 |28 |25 |04B |1143293 |1898684 |2019|05/01/2020 03:48:05 PM|41.878026693|-87.749328671|(41.878026693, -87.749328671)|1 |\n", + "|11766889|JC359918 |2019-07-17 15:07:00|024XX W FOSTER AVE |0610|BURGLARY |FORCIBLE ENTRY |RESIDENCE - PORCH / HALLWAY |true |false |2011|020 |40 |4 |05 |1158877 |1934454 |2019|05/01/2020 03:48:05 PM|41.975876871|-87.691123958|(41.975876871, -87.691123958)|1 |\n", + "|11740807|JC328262 |2019-06-30 05:04:00|048XX W AUGUSTA BLVD |041A|BATTERY |AGGRAVATED - HANDGUN |SIDEWALK |false |false |1531|015 |37 |25 |04B |1143848 |1906208 |2019|05/01/2020 03:48:05 PM|41.898663056|-87.74710208 |(41.898663056, -87.74710208) |1 |\n", + "|11718084|JC300874 |2019-05-13 13:00:00|062XX S EMERALD DR |1750|OFFENSE INVOLVING CHILDREN|CHILD ABUSE |GOVERNMENT BUILDING / PROPERTY |false |true |0711|007 |16 |68 |08B |1172179 |1863746 |2019|05/01/2020 03:48:05 PM|41.781565307|-87.644295268|(41.781565307, -87.644295268)|1 |\n", + "|11647696|JC215453 |2019-04-07 17:56:00|062XX S FRANCISCO AVE |041A|BATTERY |AGGRAVATED - HANDGUN |ALLEY |false |false |0823|008 |16 |66 |04B |1158121 |1863074 |2019|05/01/2020 03:48:05 PM|41.780018807|-87.69585372 |(41.780018807, -87.69585372) |1 |\n", + "|12039931|JD220113 |2019-10-18 08:00:00|120XX S LAFLIN ST |2826|OTHER OFFENSE |HARASSMENT BY ELECTRONIC MEANS |RESIDENCE |false |false |0524|005 |34 |53 |26 |null |null |2019|04/30/2020 03:50:06 PM|null |null |null |1 |\n", + "|12039974|JD220130 |2019-12-24 23:00:00|049XX N ALBANY AVE |0890|THEFT |FROM BUILDING |RESIDENCE |false |false |1713|017 |33 |14 |06 |null |null |2019|04/30/2020 03:50:06 PM|null |null |null |1 |\n", + "|12039885|JD219909 |2019-10-01 02:00:00|021XX S HARDING AVE |1752|OFFENSE INVOLVING CHILDREN|AGGRAVATED CRIMINAL SEXUAL ABUSE BY FAMILY MEMBER|RESIDENCE |false |true |1014|010 |24 |29 |17 |null |null |2019|04/30/2020 03:50:06 PM|null |null |null |1 |\n", + "|12039884|JD220045 |2019-07-01 12:00:00|026XX W BALMORAL AVE |1153|DECEPTIVE PRACTICE |FINANCIAL IDENTITY THEFT OVER $ 300 |RESIDENCE |false |false |2011|020 |40 |4 |11 |null |null |2019|04/30/2020 03:50:06 PM|null |null |null |1 |\n", + "|12038367|JD218291 |2019-04-21 12:00:00|071XX S CYRIL AVE |0890|THEFT |FROM BUILDING |APARTMENT |false |false |0333|003 |5 |43 |06 |1190602 |1857828 |2019|04/30/2020 03:47:59 PM|41.764900898|-87.576943938|(41.764900898, -87.576943938)|1 |\n", + "|12038544|JD218327 |2019-12-22 13:00:00|028XX S CHRISTIANA AVE |5002|OTHER OFFENSE |OTHER VEHICLE OFFENSE |STREET |false |false |1032|010 |22 |30 |26 |1154495 |1884891 |2019|04/30/2020 03:47:59 PM|41.83996062 |-87.708565824|(41.83996062, -87.708565824) |1 |\n", + "|12038308|JD218196 |2019-07-22 08:00:00|052XX S ARCHER AVE |0910|MOTOR VEHICLE THEFT |AUTOMOBILE |PARKING LOT / GARAGE (NON RESIDENTIAL)|false |false |0815|008 |23 |57 |07 |1146707 |1870103 |2019|04/30/2020 03:47:59 PM|41.799532089|-87.737521099|(41.799532089, -87.737521099)|1 |\n", + "|12040067|JD220150 |2019-12-15 18:00:00|057XX W ROOSEVELT RD |0265|CRIMINAL SEXUAL ASSAULT |AGGRAVATED - OTHER |HOSPITAL BUILDING / GROUNDS |false |false |1513|015 |29 |25 |02 |null |null |2019|04/30/2020 03:50:06 PM|null |null |null |1 |\n", + "|12039858|JD219964 |2019-09-03 09:00:00|003XX S SPRINGFIELD AVE |1154|DECEPTIVE PRACTICE |FINANCIAL IDENTITY THEFT $300 AND UNDER |APARTMENT |false |false |1133|011 |28 |26 |11 |null |null |2019|04/30/2020 03:50:06 PM|null |null |null |1 |\n", + "|12038342|JD218257 |2019-05-01 00:00:00|002XX N MENARD AVE |1154|DECEPTIVE PRACTICE |FINANCIAL IDENTITY THEFT $300 AND UNDER |RESIDENCE |false |false |1512|015 |29 |25 |11 |1137672 |1901055 |2019|04/30/2020 03:47:59 PM|41.884636133|-87.769910804|(41.884636133, -87.769910804)|1 |\n", + "|12038644|JD218194 |2019-06-01 13:45:00|003XX N MENARD AVE |1153|DECEPTIVE PRACTICE |FINANCIAL IDENTITY THEFT OVER $ 300 |null |false |false |1512|015 |29 |25 |11 |1137654 |1901595 |2019|04/30/2020 03:47:59 PM|41.886118287|-87.769963886|(41.886118287, -87.769963886)|1 |\n", + "+--------+-----------+-------------------+-------------------------+----+--------------------------+-------------------------------------------------+--------------------------------------+------+--------+----+--------+----+--------------+--------+------------+------------+----+----------------------+------------+-------------+-----------------------------+----+\n", + "only showing top 20 rows\n", + "\n" + ], + "name": "stdout" + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "4CDifVC2Cnml", + "colab_type": "text" + }, + "source": [ + "\n", + "#### Grouping By Column" + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "M1ek2opVfqea", + "colab_type": "code", + "outputId": "b0715ee8-79df-4559-8d4f-be796bb8ddaf", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 119 + } + }, + "source": [ + "#Group By a column in PySpark\n", + "df.groupBy('Year').count().show(5)" + ], + "execution_count": 0, + "outputs": [ + { + "output_type": "stream", + "text": [ + "+----+------+\n", + "|Year| count|\n", + "+----+------+\n", + "|2019|259013|\n", + "+----+------+\n", + "\n" + ], + "name": "stdout" + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "CbpEj9fECrW3", + "colab_type": "text" + }, + "source": [ + "\n", + "#### Removing a Column" + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "xsb9PXxpfnmh", + "colab_type": "code", + "outputId": "d9601326-b08f-4367-cf61-5b5da0d5c9e4", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 224 + } + }, + "source": [ + "#Remove columns in PySpark\n", + "df = df.drop('Test')\n", + "df.show(5,truncate=False)" + ], + "execution_count": 0, + "outputs": [ + { + "output_type": "stream", + "text": [ + "+--------+-----------+-------------------+-------------------------+----+------------------+-----------------------------------+--------------------+------+--------+----+--------+----+--------------+--------+------------+------------+----+----------------------+------------+-------------+-----------------------------+\n", + "|ID |Case Number|Date |Block |IUCR|Primary Type |Description |Location Description|Arrest|Domestic|Beat|District|Ward|Community Area|FBI Code|X Coordinate|Y Coordinate|Year|Updated On |Latitude |Longitude |Location |\n", + "+--------+-----------+-------------------+-------------------------+----+------------------+-----------------------------------+--------------------+------+--------+----+--------+----+--------------+--------+------------+------------+----+----------------------+------------+-------------+-----------------------------+\n", + "|12040452|JD220630 |2019-11-21 12:00:00|004XX N WABASH AVE |1153|DECEPTIVE PRACTICE|FINANCIAL IDENTITY THEFT OVER $ 300|RESIDENCE |false |false |1834|018 |42 |8 |11 |null |null |2019|05/01/2020 03:49:55 PM|null |null |null |\n", + "|12039199|JD219081 |2019-04-01 08:00:00|021XX S INDIANA AVE |1153|DECEPTIVE PRACTICE|FINANCIAL IDENTITY THEFT OVER $ 300|RESIDENCE |false |false |0132|001 |3 |33 |11 |1177977 |1890072 |2019|05/01/2020 03:48:05 PM|41.853676611|-87.622239838|(41.853676611, -87.622239838)|\n", + "|12040466|JD220650 |2019-11-07 12:00:00|026XX N NEW ENGLAND AVE |1152|DECEPTIVE PRACTICE|ILLEGAL USE CASH CARD |RESIDENCE |false |false |2512|025 |36 |18 |11 |null |null |2019|05/01/2020 03:49:55 PM|null |null |null |\n", + "|11938499|JD100565 |2019-12-31 11:30:00|070XX S COTTAGE GROVE AVE|0460|BATTERY |SIMPLE |OTHER (SPECIFY) |true |false |0321|003 |6 |42 |08B |1182773 |1858454 |2019|05/01/2020 03:48:05 PM|41.76680404 |-87.605619683|(41.76680404, -87.605619683) |\n", + "|11864640|JC477069 |2019-10-18 09:56:00|002XX S LAVERGNE AVE |041A|BATTERY |AGGRAVATED - HANDGUN |SIDEWALK |false |false |1533|015 |28 |25 |04B |1143293 |1898684 |2019|05/01/2020 03:48:05 PM|41.878026693|-87.749328671|(41.878026693, -87.749328671)|\n", + "+--------+-----------+-------------------+-------------------------+----+------------------+-----------------------------------+--------------------+------+--------+----+--------+----+--------------+--------+------------+------------+----+----------------------+------------+-------------+-----------------------------+\n", + "only showing top 5 rows\n", + "\n" + ], + "name": "stdout" + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "WbKK5iHwmIoV", + "colab_type": "text" + }, + "source": [ + "\n", + "# Working with Rows" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "9bKlvX-SH-Wy", + "colab_type": "text" + }, + "source": [ + "\n", + "#### Filtering Rows" + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "YNfcjOIknA3n", + "colab_type": "code", + "outputId": "52be7cc5-4aa2-4264-f9d2-e0aa9df1d162", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 479 + } + }, + "source": [ + "# Filtering rows in PySpark\n", + "df.filter(col('Date')<'2019-06-01').show(truncate=False)" + ], + "execution_count": 0, + "outputs": [ + { + "output_type": "stream", + "text": [ + "+--------+-----------+-------------------+----------------------+----+--------------------------+-------------------------------------------------------+------------------------------------------+------+--------+----+--------+----+--------------+--------+------------+------------+----+----------------------+------------+-------------+-----------------------------+\n", + "|ID |Case Number|Date |Block |IUCR|Primary Type |Description |Location Description |Arrest|Domestic|Beat|District|Ward|Community Area|FBI Code|X Coordinate|Y Coordinate|Year|Updated On |Latitude |Longitude |Location |\n", + "+--------+-----------+-------------------+----------------------+----+--------------------------+-------------------------------------------------------+------------------------------------------+------+--------+----+--------+----+--------------+--------+------------+------------+----+----------------------+------------+-------------+-----------------------------+\n", + "|12039199|JD219081 |2019-04-01 08:00:00|021XX S INDIANA AVE |1153|DECEPTIVE PRACTICE |FINANCIAL IDENTITY THEFT OVER $ 300 |RESIDENCE |false |false |0132|001 |3 |33 |11 |1177977 |1890072 |2019|05/01/2020 03:48:05 PM|41.853676611|-87.622239838|(41.853676611, -87.622239838)|\n", + "|11718084|JC300874 |2019-05-13 13:00:00|062XX S EMERALD DR |1750|OFFENSE INVOLVING CHILDREN|CHILD ABUSE |GOVERNMENT BUILDING / PROPERTY |false |true |0711|007 |16 |68 |08B |1172179 |1863746 |2019|05/01/2020 03:48:05 PM|41.781565307|-87.644295268|(41.781565307, -87.644295268)|\n", + "|11647696|JC215453 |2019-04-07 17:56:00|062XX S FRANCISCO AVE |041A|BATTERY |AGGRAVATED - HANDGUN |ALLEY |false |false |0823|008 |16 |66 |04B |1158121 |1863074 |2019|05/01/2020 03:48:05 PM|41.780018807|-87.69585372 |(41.780018807, -87.69585372) |\n", + "|12038367|JD218291 |2019-04-21 12:00:00|071XX S CYRIL AVE |0890|THEFT |FROM BUILDING |APARTMENT |false |false |0333|003 |5 |43 |06 |1190602 |1857828 |2019|04/30/2020 03:47:59 PM|41.764900898|-87.576943938|(41.764900898, -87.576943938)|\n", + "|12038342|JD218257 |2019-05-01 00:00:00|002XX N MENARD AVE |1154|DECEPTIVE PRACTICE |FINANCIAL IDENTITY THEFT $300 AND UNDER |RESIDENCE |false |false |1512|015 |29 |25 |11 |1137672 |1901055 |2019|04/30/2020 03:47:59 PM|41.884636133|-87.769910804|(41.884636133, -87.769910804)|\n", + "|11994131|JD167522 |2019-01-01 09:00:00|013XX W 110TH PL |1195|DECEPTIVE PRACTICE |FINANCIAL EXPLOITATION OF AN ELDERLY OR DISABLED PERSON|APARTMENT |false |false |2234|022 |34 |75 |11 |1169349 |1831489 |2019|04/30/2020 03:47:59 PM|41.693109292|-87.655602002|(41.693109292, -87.655602002)|\n", + "|11696379|JC273783 |2019-05-22 14:40:00|029XX S FORT DEARBORN |041A|BATTERY |AGGRAVATED - HANDGUN |PARKING LOT / GARAGE (NON RESIDENTIAL) |false |false |0133|001 |4 |35 |04B |1181530 |1885736 |2019|04/30/2020 03:47:59 PM|41.841696878|-87.609333378|(41.841696878, -87.609333378)|\n", + "|11625478|JC188359 |2019-03-16 20:39:00|068XX N OVERHILL AVE |0610|BURGLARY |FORCIBLE ENTRY |RESIDENCE |true |false |1612|016 |41 |9 |05 |1123909 |1944430 |2019|04/30/2020 03:47:59 PM|42.003899378|-87.819496765|(42.003899378, -87.819496765)|\n", + "|12036474|JD216160 |2019-04-05 10:00:00|003XX W HUBBARD ST |0810|THEFT |OVER $500 |APARTMENT |false |false |1831|018 |42 |8 |06 |1173879 |1903256 |2019|04/29/2020 03:53:17 PM|41.889946527|-87.63688824 |(41.889946527, -87.63688824) |\n", + "|11566441|JC116418 |2019-01-14 03:00:00|036XX N WHIPPLE ST |0265|CRIMINAL SEXUAL ASSAULT |AGGRAVATED - OTHER |APARTMENT |true |false |1733|017 |33 |16 |02 |1155406 |1924019 |2019|04/29/2020 03:53:17 PM|41.947313291|-87.704169938|(41.947313291, -87.704169938)|\n", + "|11640117|JC205456 |2019-03-27 09:00:00|082XX S LAFLIN ST |0281|CRIMINAL SEXUAL ASSAULT |NON-AGGRAVATED |RESIDENCE |false |true |0614|006 |21 |71 |02 |1167770 |1850079 |2019|04/28/2020 03:45:44 PM|41.744157062|-87.660851418|(41.744157062, -87.660851418)|\n", + "|11825583|JC430037 |2019-04-02 06:00:00|0000X S CONCOURSE A ST|0810|THEFT |OVER $500 |AIRPORT TERMINAL UPPER LEVEL - SECURE AREA|true |false |0813|008 |23 |56 |06 |1145563 |1865318 |2019|04/26/2020 03:45:32 PM|41.786422961|-87.741837247|(41.786422961, -87.741837247)|\n", + "|11699523|JC278066 |2019-05-25 12:49:00|028XX W 51ST ST |031A|ROBBERY |ARMED - HANDGUN |ALLEY |true |false |0923|009 |14 |63 |03 |1158131 |1870706 |2019|04/26/2020 03:45:32 PM|41.800961869|-87.695609509|(41.800961869, -87.695609509)|\n", + "|11677547|JC250749 |2019-05-05 13:36:00|008XX E 75TH ST |051A|ASSAULT |AGGRAVATED - HANDGUN |SIDEWALK |false |false |0323|003 |8 |69 |04A |1182958 |1855461 |2019|04/26/2020 03:45:32 PM|41.758586652|-87.605034476|(41.758586652, -87.605034476)|\n", + "|12035312|JD214438 |2019-03-29 08:00:00|049XX W ARTHINGTON ST |1153|DECEPTIVE PRACTICE |FINANCIAL IDENTITY THEFT OVER $ 300 |null |false |false |1533|015 |29 |25 |11 |1143756 |1895516 |2019|04/25/2020 03:46:57 PM|41.869324647|-87.747707988|(41.869324647, -87.747707988)|\n", + "|12035047|JD214561 |2019-05-01 00:00:00|002XX W 24TH PL |1153|DECEPTIVE PRACTICE |FINANCIAL IDENTITY THEFT OVER $ 300 |RESIDENCE |false |false |0914|009 |25 |34 |11 |1175108 |1888066 |2019|04/25/2020 03:46:57 PM|41.848236712|-87.632830074|(41.848236712, -87.632830074)|\n", + "|11618719|JC180138 |2019-03-10 10:54:00|010XX S CLARK ST |0479|BATTERY |AGGRAVATED - HANDS, FISTS, FEET, SERIOUS INJURY |RESIDENCE |true |true |0123|001 |4 |32 |04B |1175680 |1895895 |2019|04/25/2020 03:46:57 PM|41.869707206|-87.630495617|(41.869707206, -87.630495617)|\n", + "|12034436|JD211780 |2019-02-19 09:00:00|055XX W WILSON AVE |1153|DECEPTIVE PRACTICE |FINANCIAL IDENTITY THEFT OVER $ 300 |APARTMENT |false |false |1623|016 |45 |15 |11 |1138670 |1930193 |2019|04/24/2020 03:45:07 PM|41.96457596 |-87.765537442|(41.96457596, -87.765537442) |\n", + "|11700425|JC279076 |2019-05-26 06:09:00|056XX W NORTH AVE |041A|BATTERY |AGGRAVATED - HANDGUN |RESTAURANT |false |false |2531|025 |29 |25 |04B |1138594 |1910075 |2019|04/24/2020 03:45:07 PM|41.909371462|-87.766306125|(41.909371462, -87.766306125)|\n", + "|11586737|JC140804 |2019-02-04 23:00:00|002XX N WABASH AVE |0281|CRIMINAL SEXUAL ASSAULT |NON-AGGRAVATED |HOTEL / MOTEL |false |false |0111|001 |42 |32 |02 |1176777 |1901808 |2019|04/24/2020 03:45:07 PM|41.885908101|-87.626289429|(41.885908101, -87.626289429)|\n", + "+--------+-----------+-------------------+----------------------+----+--------------------------+-------------------------------------------------------+------------------------------------------+------+--------+----+--------+----+--------------+--------+------------+------------+----+----------------------+------------+-------------+-----------------------------+\n", + "only showing top 20 rows\n", + "\n" + ], + "name": "stdout" + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "zLU-a4auIEvh", + "colab_type": "text" + }, + "source": [ + "\n", + "#### Get Distinct Rows" + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "B1RKg1UrmBQz", + "colab_type": "code", + "outputId": "083fce9b-7a63-4d69-e8d1-3acca65d6433", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 119 + } + }, + "source": [ + "#Get Unique Rows in PySpark\n", + "df.select('Year').distinct().show()" + ], + "execution_count": 0, + "outputs": [ + { + "output_type": "stream", + "text": [ + "+----+\n", + "|Year|\n", + "+----+\n", + "|2019|\n", + "+----+\n", + "\n" + ], + "name": "stdout" + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "-069UYUwIIYI", + "colab_type": "text" + }, + "source": [ + "\n", + "#### Sorting Rows" + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "4ZpeJvz0nkBI", + "colab_type": "code", + "outputId": "846c33eb-a7a4-4a57-a67d-7bf787a06bab", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 479 + } + }, + "source": [ + "# Sort Rows in PySpark\n", + "# By default the data will be sorted in ascending order\n", + "df.orderBy('Date').show(truncate=False) " + ], + "execution_count": 0, + "outputs": [ + { + "output_type": "stream", + "text": [ + "+--------+-----------+-------------------+----------------------+----+--------------------------+-----------------------------------+------------------------------+------+--------+----+--------+----+--------------+--------+------------+------------+----+----------------------+------------+-------------+-----------------------------+\n", + "|ID |Case Number|Date |Block |IUCR|Primary Type |Description |Location Description |Arrest|Domestic|Beat|District|Ward|Community Area|FBI Code|X Coordinate|Y Coordinate|Year|Updated On |Latitude |Longitude |Location |\n", + "+--------+-----------+-------------------+----------------------+----+--------------------------+-----------------------------------+------------------------------+------+--------+----+--------+----+--------------+--------+------------+------------+----+----------------------+------------+-------------+-----------------------------+\n", + "|11552674|JC100085 |2019-01-01 00:00:00|092XX S NORMAL AVE |0910|MOTOR VEHICLE THEFT |AUTOMOBILE |STREET |false |false |2223|022 |21 |73 |07 |1174586 |1843723 |2019|01/10/2019 03:16:50 PM|41.726566477|-87.636065622|(41.726566477, -87.636065622)|\n", + "|11560011|JC108820 |2019-01-01 00:00:00|068XX S OGLESBY AVE |0820|THEFT |$500 AND UNDER |STREET |false |false |0331|003 |5 |43 |06 |1192946 |1860098 |2019|01/10/2019 03:16:50 PM|41.771073064|-87.568278663|(41.771073064, -87.568278663)|\n", + "|11730841|JC315506 |2019-01-01 00:00:00|007XX E 95TH ST |1563|SEX OFFENSE |CRIMINAL SEXUAL ABUSE |DAY CARE CENTER |false |false |0633|006 |9 |49 |17 |1182808 |1842138 |2019|06/30/2019 03:56:27 PM|41.722030342|-87.605996889|(41.722030342, -87.605996889)|\n", + "|11718021|JC300596 |2019-01-01 00:00:00|102XX W ZEMKE RD |0910|MOTOR VEHICLE THEFT |AUTOMOBILE |PARKING LOT/GARAGE(NON.RESID.)|false |false |1654|016 |41 |76 |07 |1106955 |1941060 |2019|06/30/2019 03:56:27 PM|41.994913946|-87.881937669|(41.994913946, -87.881937669)|\n", + "|11714518|JC294379 |2019-01-01 00:00:00|0000X W 115TH ST |1210|DECEPTIVE PRACTICE |THEFT OF LABOR/SERVICES |APARTMENT |false |false |0522|005 |34 |53 |11 |1178223 |1828722 |2019|06/30/2019 03:56:27 PM|41.685320047|-87.623196127|(41.685320047, -87.623196127)|\n", + "|11682859|JC256913 |2019-01-01 00:00:00|074XX S HARVARD AVE |1753|OFFENSE INVOLVING CHILDREN|SEX ASSLT OF CHILD BY FAM MBR |RESIDENCE |false |true |0731|007 |6 |69 |02 |1175242 |1855669 |2019|06/12/2019 04:05:16 PM|41.759333204|-87.633306659|(41.759333204, -87.633306659)|\n", + "|11552667|JC100123 |2019-01-01 00:00:00|004XX N STATE ST |0890|THEFT |FROM BUILDING |RESTAURANT |false |false |1831|018 |42 |8 |06 |1176302 |1903096 |2019|01/10/2019 03:16:50 PM|41.889453169|-87.627994833|(41.889453169, -87.627994833)|\n", + "|11723841|JC300604 |2019-01-01 00:00:00|102XX W ZEMKE RD |0910|MOTOR VEHICLE THEFT |AUTOMOBILE |PARKING LOT/GARAGE(NON.RESID.)|false |false |1654|016 |41 |76 |07 |1106955 |1941060 |2019|06/30/2019 03:56:27 PM|41.994913946|-87.881937669|(41.994913946, -87.881937669)|\n", + "|11706371|JC286255 |2019-01-01 00:00:00|116XX S ADA ST |1544|SEX OFFENSE |SEXUAL EXPLOITATION OF A CHILD |RESIDENCE |false |false |0524|005 |34 |53 |17 |1169420 |1827617 |2019|06/30/2019 03:56:27 PM|41.68248235 |-87.655453609|(41.68248235, -87.655453609) |\n", + "|11552709|JC100020 |2019-01-01 00:00:00|044XX S WASHTENAW AVE |0486|BATTERY |DOMESTIC BATTERY SIMPLE |APARTMENT |false |true |0922|009 |15 |58 |08B |1159112 |1875020 |2019|01/10/2019 03:16:50 PM|41.812780011|-87.691893746|(41.812780011, -87.691893746)|\n", + "|11552758|JC100058 |2019-01-01 00:00:00|063XX S MARSHFIELD AVE|1310|CRIMINAL DAMAGE |TO PROPERTY |APARTMENT |false |false |0725|007 |16 |67 |14 |1166414 |1862607 |2019|01/10/2019 03:16:50 PM|41.77856457 |-87.665463557|(41.77856457, -87.665463557) |\n", + "|11558163|JC106702 |2019-01-01 00:00:00|002XX W ONTARIO ST |0890|THEFT |FROM BUILDING |BAR OR TAVERN |false |false |1831|018 |42 |8 |06 |1174469 |1904439 |2019|01/10/2019 03:16:50 PM|41.893179585|-87.634686145|(41.893179585, -87.634686145)|\n", + "|11553168|JC100745 |2019-01-01 00:00:00|008XX N MICHIGAN AVE |0890|THEFT |FROM BUILDING |RESTAURANT |false |false |1833|018 |2 |8 |06 |1177330 |1906499 |2019|01/10/2019 03:16:50 PM|41.898767916|-87.624116333|(41.898767916, -87.624116333)|\n", + "|11553381|JC100934 |2019-01-01 00:00:00|022XX N LEAMINGTON AVE|1320|CRIMINAL DAMAGE |TO VEHICLE |RESIDENCE-GARAGE |false |false |2522|025 |36 |19 |14 |1141689 |1914382 |2019|01/10/2019 03:16:50 PM|41.921133632|-87.75482958 |(41.921133632, -87.75482958) |\n", + "|11553495|JC101115 |2019-01-01 00:00:00|047XX N RACINE AVE |0281|CRIM SEXUAL ASSAULT |NON-AGGRAVATED |OTHER |false |false |1913|019 |46 |3 |02 |1167451 |1931818 |2019|01/10/2019 03:16:50 PM|41.968462892|-87.659670442|(41.968462892, -87.659670442)|\n", + "|11553914|JC101459 |2019-01-01 00:00:00|062XX W BELMONT AVE |1310|CRIMINAL DAMAGE |TO PROPERTY |OTHER |false |false |2511|025 |36 |19 |14 |1134192 |1920614 |2019|01/10/2019 03:16:50 PM|41.938370433|-87.782228587|(41.938370433, -87.782228587)|\n", + "|11556297|JC104365 |2019-01-01 00:00:00|083XX S YATES BLVD |2826|OTHER OFFENSE |HARASSMENT BY ELECTRONIC MEANS |RESIDENCE |false |false |0412|004 |7 |46 |26 |1193625 |1850166 |2019|01/10/2019 03:16:50 PM|41.74380228 |-87.566114429|(41.74380228, -87.566114429) |\n", + "|11563351|JC112682 |2019-01-01 00:00:00|045XX S EVANS AVE |2825|OTHER OFFENSE |HARASSMENT BY TELEPHONE |APARTMENT |false |true |0221|002 |4 |38 |26 |1181911 |1875136 |2019|01/18/2019 09:37:14 AM|41.812600904|-87.608263541|(41.812600904, -87.608263541)|\n", + "|11566200|JC116315 |2019-01-01 00:00:00|072XX S EAST END AVE |0320|ROBBERY |STRONGARM - NO WEAPON |APARTMENT |false |true |0324|003 |7 |43 |03 |1188788 |1857344 |2019|01/18/2019 09:37:14 AM|41.763616341|-87.58360812 |(41.763616341, -87.58360812) |\n", + "|11571975|JC123039 |2019-01-01 00:00:00|0000X N LOCKWOOD AVE |1153|DECEPTIVE PRACTICE |FINANCIAL IDENTITY THEFT OVER $ 300|RESIDENCE |false |false |1522|015 |28 |25 |11 |1141053 |1899725 |2019|01/23/2019 04:13:37 PM|41.880924861|-87.757527885|(41.880924861, -87.757527885)|\n", + "+--------+-----------+-------------------+----------------------+----+--------------------------+-----------------------------------+------------------------------+------+--------+----+--------+----+--------------+--------+------------+------------+----+----------------------+------------+-------------+-----------------------------+\n", + "only showing top 20 rows\n", + "\n" + ], + "name": "stdout" + } + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "v1CEwofMJV-D", + "colab_type": "code", + "outputId": "91c8d3c2-d1c6-4195-abbc-c540519c23c1", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 479 + } + }, + "source": [ + "# To change the sorting order, you can use the asceding parameter\n", + "df.orderBy('Date', ascending=False).show(truncate=False) " + ], + "execution_count": 0, + "outputs": [ + { + "output_type": "stream", + "text": [ + "+--------+-----------+-------------------+----------------------+----+-----------------+--------------------------------------------------+----------------------+------+--------+----+--------+----+--------------+--------+------------+------------+----+----------------------+------------+-------------+-----------------------------+\n", + "|ID |Case Number|Date |Block |IUCR|Primary Type |Description |Location Description |Arrest|Domestic|Beat|District|Ward|Community Area|FBI Code|X Coordinate|Y Coordinate|Year|Updated On |Latitude |Longitude |Location |\n", + "+--------+-----------+-------------------+----------------------+----+-----------------+--------------------------------------------------+----------------------+------+--------+----+--------+----+--------------+--------+------------+------------+----+----------------------+------------+-------------+-----------------------------+\n", + "|11938228|JD100017 |2019-12-31 23:55:00|0000X W 69TH ST |143A|WEAPONS VIOLATION|UNLAWFUL POSS OF HANDGUN |STREET |true |false |0731|007 |6 |69 |15 |1176896 |1859260 |2019|01/07/2020 03:52:21 PM|41.769150218|-87.627136786|(41.769150218, -87.627136786)|\n", + "|11940078|JD100016 |2019-12-31 23:54:00|063XX S MAY ST |0420|BATTERY |AGGRAVATED:KNIFE/CUTTING INSTR |SIDEWALK |false |false |0724|007 |16 |68 |04B |1169736 |1862855 |2019|01/08/2020 03:47:27 PM|41.779173667|-87.653277703|(41.779173667, -87.653277703)|\n", + "|11938857|JD100599 |2019-12-31 23:50:00|004XX N Ashland ave |0820|THEFT |$500 AND UNDER |BAR OR TAVERN |false |false |1215|012 |27 |24 |06 |null |null |2019|01/07/2020 03:52:21 PM|null |null |null |\n", + "|11938240|JD100002 |2019-12-31 23:48:00|004XX S CICERO AVE |143A|WEAPONS VIOLATION|UNLAWFUL POSS OF HANDGUN |VEHICLE NON-COMMERCIAL|true |false |1533|015 |29 |25 |15 |1144466 |1897452 |2019|01/07/2020 03:52:21 PM|41.874623951|-87.745052647|(41.874623951, -87.745052647)|\n", + "|11937967|JC567053 |2019-12-31 23:46:00|034XX W JACKSON BLVD |143A|WEAPONS VIOLATION|UNLAWFUL POSS OF HANDGUN |STREET |false |false |1133|011 |28 |27 |15 |1153587 |1898480 |2019|01/07/2020 03:52:21 PM|41.877268465|-87.711536692|(41.877268465, -87.711536692)|\n", + "|11938124|JD100001 |2019-12-31 23:37:00|012XX W 99TH ST |5112|OTHER OFFENSE |GUN OFFENDER: DUTY TO REPORT CHANGE OF INFORMATION|STREET |true |false |2232|022 |34 |73 |26 |1170052 |1839142 |2019|01/07/2020 03:52:21 PM|41.714095115|-87.652806763|(41.714095115, -87.652806763)|\n", + "|11937990|JC567039 |2019-12-31 23:31:00|006XX W 47TH ST |0460|BATTERY |SIMPLE |APARTMENT |false |false |0925|009 |11 |61 |08B |1172817 |1873724 |2019|01/07/2020 03:52:21 PM|41.808931947|-87.641661883|(41.808931947, -87.641661883)|\n", + "|11937983|JC567043 |2019-12-31 23:30:00|049XX N CHRISTIANA AVE|051A|ASSAULT |AGGRAVATED: HANDGUN |STREET |false |false |1713|017 |33 |14 |04A |1153158 |1932519 |2019|01/07/2020 03:52:21 PM|41.970682839|-87.712206567|(41.970682839, -87.712206567)|\n", + "|11937963|JC567049 |2019-12-31 23:21:00|101XX S NORMAL AVE |5111|OTHER OFFENSE |GUN OFFENDER: ANNUAL REGISTRATION |STREET |true |false |2232|022 |9 |73 |26 |1174759 |1837501 |2019|01/07/2020 03:52:21 PM|41.709488593|-87.635616613|(41.709488593, -87.635616613)|\n", + "|11939218|JD100095 |2019-12-31 23:15:00|001XX N CLARK ST |0810|THEFT |OVER $500 |STREET |false |false |0111|001 |42 |32 |06 |1175507 |1901535 |2019|01/07/2020 03:52:21 PM|41.885187593|-87.630961294|(41.885187593, -87.630961294)|\n", + "|11938244|JC567030 |2019-12-31 23:15:00|070XX S CHAPPEL AVE |0486|BATTERY |DOMESTIC BATTERY SIMPLE |APARTMENT |false |true |0331|003 |5 |43 |08B |1191096 |1858692 |2019|01/07/2020 03:52:21 PM|41.767259848|-87.575105403|(41.767259848, -87.575105403)|\n", + "|11938338|JC567037 |2019-12-31 23:15:00|001XX W 68TH ST |0486|BATTERY |DOMESTIC BATTERY SIMPLE |APARTMENT |false |true |0722|007 |6 |69 |08B |1176673 |1859896 |2019|01/07/2020 03:52:21 PM|41.770900494|-87.627935073|(41.770900494, -87.627935073)|\n", + "|11937952|JC567038 |2019-12-31 23:14:00|007XX N MICHIGAN AVE |0486|BATTERY |DOMESTIC BATTERY SIMPLE |HOTEL/MOTEL |false |true |1834|018 |42 |8 |08B |1177303 |1905177 |2019|01/07/2020 03:52:21 PM|41.895140898|-87.624255632|(41.895140898, -87.624255632)|\n", + "|11938039|JC567055 |2019-12-31 23:14:00|046XX S DAMEN AVE |143A|WEAPONS VIOLATION|UNLAWFUL POSS OF HANDGUN |STREET |true |false |0924|009 |12 |61 |15 |1163735 |1873901 |2019|01/07/2020 03:52:21 PM|41.809613401|-87.674967882|(41.809613401, -87.674967882)|\n", + "|11938007|JC567034 |2019-12-31 23:12:00|030XX W 21ST PL |0486|BATTERY |DOMESTIC BATTERY SIMPLE |RESIDENCE |true |true |1022|010 |24 |30 |08B |1156456 |1889560 |2019|01/07/2020 03:52:21 PM|41.852733523|-87.701243634|(41.852733523, -87.701243634)|\n", + "|11937955|JC567056 |2019-12-31 23:12:00|080XX S KINGSTON AVE |0486|BATTERY |DOMESTIC BATTERY SIMPLE |APARTMENT |false |true |0422|004 |7 |46 |08B |1194583 |1851989 |2019|01/07/2020 03:52:21 PM|41.748781248|-87.562544473|(41.748781248, -87.562544473)|\n", + "|11938003|JC567031 |2019-12-31 23:10:00|100XX S LAFAYETTE AVE |0486|BATTERY |DOMESTIC BATTERY SIMPLE |RESIDENCE |false |true |0511|005 |9 |49 |08B |1177696 |1838416 |2019|01/07/2020 03:52:21 PM|41.711933671|-87.624833412|(41.711933671, -87.624833412)|\n", + "|11938034|JC567024 |2019-12-31 23:06:00|036XX S STATE ST |143A|WEAPONS VIOLATION|UNLAWFUL POSS OF HANDGUN |STREET |true |false |0213|002 |3 |35 |15 |1176895 |1880755 |2019|01/07/2020 03:52:21 PM|41.828134581|-87.62649256 |(41.828134581, -87.62649256) |\n", + "|11937958|JC567042 |2019-12-31 23:02:00|014XX E 76TH ST |0460|BATTERY |SIMPLE |RESIDENCE |true |false |0411|004 |5 |43 |08B |1187313 |1854891 |2019|01/07/2020 03:52:21 PM|41.756920236|-87.589092011|(41.756920236, -87.589092011)|\n", + "|11943133|JD105884 |2019-12-31 23:00:00|007XX N CENTRAL AVE |1320|CRIMINAL DAMAGE |TO VEHICLE |VACANT LOT/LAND |false |false |1511|015 |29 |25 |14 |1138889 |1904432 |2019|01/09/2020 03:55:23 PM|41.893881034|-87.765359656|(41.893881034, -87.765359656)|\n", + "+--------+-----------+-------------------+----------------------+----+-----------------+--------------------------------------------------+----------------------+------+--------+----+--------+----+--------------+--------+------------+------------+----+----------------------+------------+-------------+-----------------------------+\n", + "only showing top 20 rows\n", + "\n" + ], + "name": "stdout" + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "aN0-A_JsIX-X", + "colab_type": "text" + }, + "source": [ + "\n", + "#### Union Dataframes" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "VH0KOaBrJt6v", + "colab_type": "text" + }, + "source": [ + "You will see three main methods for performing union of dataframes. It is important to know the difference between them and which one is preferred:\n", + "\n", + "* `union()` – It is used to merge two DataFrames of the same structure/schema. If schemas are not the same, it returns an error\n", + "* `unionAll()` – This function is deprecated since Spark 2.0.0, and replaced with union()\n", + "* `unionByName()` - This fucntion is used to merge two dataframes based on column name.\n", + "\n", + "> Since `unionAll()` is deprecated, **`union()` is the preferred method for merging dataframes.**\n", + "
      \n", + "> The difference between `unionByName()` and `union()` is that `unionByName()` resolves columns by name, not by position.\n", + "\n", + "In other SQLs, Union eliminates the duplicates but UnionAll merges two datasets, thereby including duplicate records. But, in PySpark, both behave the same and includes duplicate records. The recommendation is to use `distinct()` or `dropDuplicates()` to remove duplicate records." + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "bCZIzfYmnx--", + "colab_type": "code", + "outputId": "5a2ddbe0-e70a-413d-c118-d2d26f6815f7", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 34 + } + }, + "source": [ + "# Append rows in PySpark.\n", + "one_day = spark.read.csv('reported-crimes.csv',header=True).withColumn('Date',to_timestamp(col('Date'),'MM/dd/yyyy hh:mm:ss a')).filter(col('Date')==lit('2019-07-30'))\n", + "df.filter(col('Date')==lit('2019-07-30')).count()" + ], + "execution_count": 0, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "10" + ] + }, + "metadata": { + "tags": [] + }, + "execution_count": 31 + } + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "k9CZlZ4Got_e", + "colab_type": "code", + "outputId": "85cd7f36-7915-4f42-9e67-d1f90c642223", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 34 + } + }, + "source": [ + "df.union(one_day).filter(col('Date')==lit('2019-07-30')).count()" + ], + "execution_count": 0, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "20" + ] + }, + "metadata": { + "tags": [] + }, + "execution_count": 32 + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "1pfPzVOFqC_8", + "colab_type": "text" + }, + "source": [ + "**Result:**\n", + "\n", + "> As you can see here, there were 6 crimes commited on 2019-07-30, and after union, there's 12 records.\n", + "\n" + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "CjWjzWBoMxx0", + "colab_type": "code", + "outputId": "60f1d420-c647-4de3-ddba-92dbed2dd75d", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 136 + } + }, + "source": [ + "# Creating two dataframes with jumbled columns\n", + "df1 = spark.createDataFrame([[1, 2, 3]], [\"col0\", \"col1\", \"col2\"])\n", + "df2 = spark.createDataFrame([[4, 5, 6]], [\"col1\", \"col2\", \"col0\"])\n", + "df1.unionByName(df2).show()" + ], + "execution_count": 0, + "outputs": [ + { + "output_type": "stream", + "text": [ + "+----+----+----+\n", + "|col0|col1|col2|\n", + "+----+----+----+\n", + "| 1| 2| 3|\n", + "| 6| 4| 5|\n", + "+----+----+----+\n", + "\n" + ], + "name": "stdout" + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "EX33t8e3PyGy", + "colab_type": "text" + }, + "source": [ + "**Result:**\n", + "\n", + "> As you can see here, the two dataframes have been successfully merged based on their column names.\n", + "\n" + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "6tVJeAY_plob", + "colab_type": "code", + "outputId": "bad14931-64f4-472e-bc4c-1144ad0b4a08", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 289 + } + }, + "source": [ + "# Top 10 number of reported crimes by Primary Type, in descending order of Occurence\n", + "df.groupBy(\"Primary Type\").count().orderBy('count', ascending=False).show(10)" + ], + "execution_count": 0, + "outputs": [ + { + "output_type": "stream", + "text": [ + "+-------------------+-----+\n", + "| Primary Type|count|\n", + "+-------------------+-----+\n", + "| THEFT|62362|\n", + "| BATTERY|49483|\n", + "| CRIMINAL DAMAGE|26666|\n", + "| ASSAULT|20605|\n", + "| DECEPTIVE PRACTICE|18233|\n", + "| OTHER OFFENSE|16682|\n", + "| NARCOTICS|14210|\n", + "| BURGLARY| 9624|\n", + "|MOTOR VEHICLE THEFT| 8979|\n", + "| ROBBERY| 7987|\n", + "+-------------------+-----+\n", + "only showing top 10 rows\n", + "\n" + ], + "name": "stdout" + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "xOQPOt19q_he", + "colab_type": "text" + }, + "source": [ + "\n", + "# Hands-on Questions 🤚 !" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "uk7x_PCarIff", + "colab_type": "text" + }, + "source": [ + "**What percentage of reported crimes resulted in an arrest?**" + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "zZkkAz-srcMf", + "colab_type": "code", + "colab": {} + }, + "source": [ + "# Answer" + ], + "execution_count": 0, + "outputs": [] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "9Qzp-Vb-ra6w", + "colab_type": "text" + }, + "source": [ + "**What are the top 3 locations for reported crimes?**" + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "qI6srWS4rYwv", + "colab_type": "code", + "colab": {} + }, + "source": [ + "# Answer" + ], + "execution_count": 0, + "outputs": [] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "aHjILb1DriuX", + "colab_type": "text" + }, + "source": [ + "\n", + "# Functions" + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "x3vlC7ZerlKb", + "colab_type": "code", + "outputId": "e7188bca-b33e-44f5-e3a1-9b1699915173", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 54 + } + }, + "source": [ + "# Functions available in PySpark\n", + "from pyspark.sql import functions\n", + "print(dir(functions))" + ], + "execution_count": 0, + "outputs": [ + { + "output_type": "stream", + "text": [ + "['Column', 'DataFrame', 'DataType', 'PandasUDFType', 'PythonEvalType', 'SparkContext', 'StringType', 'UserDefinedFunction', '__all__', '__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__spec__', '_binary_mathfunctions', '_collect_list_doc', '_collect_set_doc', '_create_binary_mathfunction', '_create_column_from_literal', '_create_function', '_create_udf', '_create_window_function', '_functions', '_functions_1_4', '_functions_1_6', '_functions_2_1', '_functions_2_4', '_functions_deprecated', '_lit_doc', '_message', '_string_functions', '_test', '_to_java_column', '_to_seq', '_window_functions', '_wrap_deprecated_function', 'abs', 'acos', 'add_months', 'approxCountDistinct', 'approx_count_distinct', 'array', 'array_contains', 'array_distinct', 'array_except', 'array_intersect', 'array_join', 'array_max', 'array_min', 'array_position', 'array_remove', 'array_repeat', 'array_sort', 'array_union', 'arrays_overlap', 'arrays_zip', 'asc', 'asc_nulls_first', 'asc_nulls_last', 'ascii', 'asin', 'atan', 'atan2', 'avg', 'base64', 'basestring', 'bin', 'bitwiseNOT', 'blacklist', 'broadcast', 'bround', 'cbrt', 'ceil', 'coalesce', 'col', 'collect_list', 'collect_set', 'column', 'concat', 'concat_ws', 'conv', 'corr', 'cos', 'cosh', 'count', 'countDistinct', 'covar_pop', 'covar_samp', 'crc32', 'create_map', 'cume_dist', 'current_date', 'current_timestamp', 'date_add', 'date_format', 'date_sub', 'date_trunc', 'datediff', 'dayofmonth', 'dayofweek', 'dayofyear', 'decode', 'degrees', 'dense_rank', 'desc', 'desc_nulls_first', 'desc_nulls_last', 'element_at', 'encode', 'exp', 'explode', 'explode_outer', 'expm1', 'expr', 'factorial', 'first', 'flatten', 'floor', 'format_number', 'format_string', 'from_json', 'from_unixtime', 'from_utc_timestamp', 'functools', 'get_json_object', 'greatest', 'grouping', 'grouping_id', 'hash', 'hex', 'hour', 'hypot', 'ignore_unicode_prefix', 'initcap', 'input_file_name', 'instr', 'isnan', 'isnull', 'json_tuple', 'kurtosis', 'lag', 'last', 'last_day', 'lead', 'least', 'length', 'levenshtein', 'lit', 'locate', 'log', 'log10', 'log1p', 'log2', 'lower', 'lpad', 'ltrim', 'map_concat', 'map_from_arrays', 'map_from_entries', 'map_keys', 'map_values', 'max', 'md5', 'mean', 'min', 'minute', 'monotonically_increasing_id', 'month', 'months_between', 'nanvl', 'next_day', 'ntile', 'pandas_udf', 'percent_rank', 'posexplode', 'posexplode_outer', 'pow', 'quarter', 'radians', 'rand', 'randn', 'rank', 'regexp_extract', 'regexp_replace', 'repeat', 'reverse', 'rint', 'round', 'row_number', 'rpad', 'rtrim', 'schema_of_json', 'second', 'sequence', 'sha1', 'sha2', 'shiftLeft', 'shiftRight', 'shiftRightUnsigned', 'shuffle', 'signum', 'sin', 'since', 'sinh', 'size', 'skewness', 'slice', 'sort_array', 'soundex', 'spark_partition_id', 'split', 'sqrt', 'stddev', 'stddev_pop', 'stddev_samp', 'struct', 'substring', 'substring_index', 'sum', 'sumDistinct', 'sys', 'tan', 'tanh', 'toDegrees', 'toRadians', 'to_date', 'to_json', 'to_timestamp', 'to_utc_timestamp', 'translate', 'trim', 'trunc', 'udf', 'unbase64', 'unhex', 'unix_timestamp', 'upper', 'var_pop', 'var_samp', 'variance', 'warnings', 'weekofyear', 'when', 'window', 'year']\n" + ], + "name": "stdout" + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "PIKigra7A34e", + "colab_type": "text" + }, + "source": [ + "\n", + "## String Functions" + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "63QDccSjBqC4", + "colab_type": "code", + "colab": {} + }, + "source": [ + "# Loading the data\n", + "from pyspark.sql.functions import col\n", + "df = spark.read.csv('reported-crimes.csv',header=True).withColumn('Date',to_timestamp(col('Date'),'MM/dd/yyyy hh:mm:ss a'))" + ], + "execution_count": 0, + "outputs": [] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "LiXWN8DUA9x6", + "colab_type": "text" + }, + "source": [ + "**Display the Primary Type column in lower and upper characters, and the first 4 characters of the column**" + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "52Gh9c99BZFr", + "colab_type": "code", + "outputId": "0746757f-4b2c-4396-c65c-07e42b17e947", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 459 + } + }, + "source": [ + "from pyspark.sql.functions import col,lower, upper, substring\n", + "help(substring)\n", + "df.select(lower(col('Primary Type')),upper(col('Primary Type')),substring(col('Primary Type'),1,4)).show(5, False)" + ], + "execution_count": 0, + "outputs": [ + { + "output_type": "stream", + "text": [ + "Help on function substring in module pyspark.sql.functions:\n", + "\n", + "substring(str, pos, len)\n", + " Substring starts at `pos` and is of length `len` when str is String type or\n", + " returns the slice of byte array that starts at `pos` in byte and is of length `len`\n", + " when str is Binary type.\n", + " \n", + " .. note:: The position is not zero based, but 1 based index.\n", + " \n", + " >>> df = spark.createDataFrame([('abcd',)], ['s',])\n", + " >>> df.select(substring(df.s, 1, 2).alias('s')).collect()\n", + " [Row(s='ab')]\n", + " \n", + " .. versionadded:: 1.5\n", + "\n", + "+-------------------+-------------------+-----------------------------+\n", + "|lower(Primary Type)|upper(Primary Type)|substring(Primary Type, 1, 4)|\n", + "+-------------------+-------------------+-----------------------------+\n", + "|deceptive practice |DECEPTIVE PRACTICE |DECE |\n", + "|deceptive practice |DECEPTIVE PRACTICE |DECE |\n", + "|deceptive practice |DECEPTIVE PRACTICE |DECE |\n", + "|battery |BATTERY |BATT |\n", + "|battery |BATTERY |BATT |\n", + "+-------------------+-------------------+-----------------------------+\n", + "only showing top 5 rows\n", + "\n" + ], + "name": "stdout" + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "ldtA0wk9BMkT", + "colab_type": "text" + }, + "source": [ + "\n", + "## Numeric functions" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "kmz4G5LVBOs6", + "colab_type": "text" + }, + "source": [ + "**Show the oldest date and the most recent date**" + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "wBDDH-YpBbdk", + "colab_type": "code", + "outputId": "e6389273-b5ec-4f47-893a-544ac45abdad", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 119 + } + }, + "source": [ + "from pyspark.sql.functions import min, max\n", + "df.select(min(col('Date')), max(col('Date'))).show()" + ], + "execution_count": 0, + "outputs": [ + { + "output_type": "stream", + "text": [ + "+-------------------+-------------------+\n", + "| min(Date)| max(Date)|\n", + "+-------------------+-------------------+\n", + "|2019-01-01 00:00:00|2019-12-31 23:55:00|\n", + "+-------------------+-------------------+\n", + "\n" + ], + "name": "stdout" + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "KQ6Ul9HGCwC3", + "colab_type": "text" + }, + "source": [ + "\n", + "## Date" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "9WFrADdkBPsX", + "colab_type": "text" + }, + "source": [ + "**What is 3 days earlier that the oldest date and 3 days later than the most recent date?**" + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "YrNmD5umAx__", + "colab_type": "code", + "outputId": "59a608be-c9e0-44cc-e2d3-d6046861059a", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 119 + } + }, + "source": [ + "from pyspark.sql.functions import date_add, date_sub\n", + "df.select(date_add(max(col('Date')),3), date_sub(min(col('Date')),3)).show()" + ], + "execution_count": 0, + "outputs": [ + { + "output_type": "stream", + "text": [ + "+----------------------+----------------------+\n", + "|date_add(max(Date), 3)|date_sub(min(Date), 3)|\n", + "+----------------------+----------------------+\n", + "| 2020-01-03| 2018-12-29|\n", + "+----------------------+----------------------+\n", + "\n" + ], + "name": "stdout" + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "sY6PstyLDp6P", + "colab_type": "text" + }, + "source": [ + "\n", + "# Working with Dates" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "s1jmBN2qFHyk", + "colab_type": "text" + }, + "source": [ + "> [PySpark follows SimpleDateFormat table of Java. Click here to view the docs.](https://docs.oracle.com/javase/7/docs/api/java/text/SimpleDateFormat.html)" + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "sCTeI_JvDCsH", + "colab_type": "code", + "outputId": "825a09aa-425c-4843-ad5c-af0dd4d0fdef", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 119 + } + }, + "source": [ + "from pyspark.sql.functions import to_date, to_timestamp, lit\n", + "df = spark.createDataFrame([('2019-12-25 13:30:00',)], ['Christmas'])\n", + "df.show()" + ], + "execution_count": 0, + "outputs": [ + { + "output_type": "stream", + "text": [ + "+-------------------+\n", + "| Christmas|\n", + "+-------------------+\n", + "|2019-12-25 13:30:00|\n", + "+-------------------+\n", + "\n" + ], + "name": "stdout" + } + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "ZH8ja1eHEW8x", + "colab_type": "code", + "outputId": "bf8216eb-a6d9-4d1d-8c03-bce2e5d57de5", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 119 + } + }, + "source": [ + "df.select(to_date(col('Christmas'),'yyyy-MM-dd HH:mm:ss'), to_timestamp(col('Christmas'),'yyyy-MM-dd HH:mm:ss')).show()" + ], + "execution_count": 0, + "outputs": [ + { + "output_type": "stream", + "text": [ + "+-------------------------------------------+------------------------------------------------+\n", + "|to_date(`Christmas`, 'yyyy-MM-dd HH:mm:ss')|to_timestamp(`Christmas`, 'yyyy-MM-dd HH:mm:ss')|\n", + "+-------------------------------------------+------------------------------------------------+\n", + "| 2019-12-25| 2019-12-25 13:30:00|\n", + "+-------------------------------------------+------------------------------------------------+\n", + "\n" + ], + "name": "stdout" + } + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "7g9m_8PPErI1", + "colab_type": "code", + "outputId": "6a2be393-b89b-4d17-a16f-e9b390097851", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 119 + } + }, + "source": [ + "df = spark.createDataFrame([('25/Dec/2019 13:30:00',)], ['Christmas'])\n", + "df.select(to_date(col('Christmas'),'dd/MMM/yyyy HH:mm:ss'), to_timestamp(col('Christmas'),'dd/MMM/yyyy HH:mm:ss')).show()" + ], + "execution_count": 0, + "outputs": [ + { + "output_type": "stream", + "text": [ + "+--------------------------------------------+-------------------------------------------------+\n", + "|to_date(`Christmas`, 'dd/MMM/yyyy HH:mm:ss')|to_timestamp(`Christmas`, 'dd/MMM/yyyy HH:mm:ss')|\n", + "+--------------------------------------------+-------------------------------------------------+\n", + "| 2019-12-25| 2019-12-25 13:30:00|\n", + "+--------------------------------------------+-------------------------------------------------+\n", + "\n" + ], + "name": "stdout" + } + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "x26Ls5SzE9qJ", + "colab_type": "code", + "outputId": "40665c93-601b-4491-b80c-b7f7527da654", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 323 + } + }, + "source": [ + "df = spark.createDataFrame([('12/25/2019 01:30:00 PM',)], ['Christmas'])\n", + "df.show(1)\n", + "df.show(1, truncate = False)\n", + "df.select(to_date(col('Christmas'),'MM/dd/yyyy hh:mm:ss aa'), to_timestamp(col('Christmas'),'MM/dd/yyyy hh:mm:ss aa')).show(truncate=False)" + ], + "execution_count": 0, + "outputs": [ + { + "output_type": "stream", + "text": [ + "+--------------------+\n", + "| Christmas|\n", + "+--------------------+\n", + "|12/25/2019 01:30:...|\n", + "+--------------------+\n", + "\n", + "+----------------------+\n", + "|Christmas |\n", + "+----------------------+\n", + "|12/25/2019 01:30:00 PM|\n", + "+----------------------+\n", + "\n", + "+----------------------------------------------+---------------------------------------------------+\n", + "|to_date(`Christmas`, 'MM/dd/yyyy hh:mm:ss aa')|to_timestamp(`Christmas`, 'MM/dd/yyyy hh:mm:ss aa')|\n", + "+----------------------------------------------+---------------------------------------------------+\n", + "|2019-12-25 |2019-12-25 13:30:00 |\n", + "+----------------------------------------------+---------------------------------------------------+\n", + "\n" + ], + "name": "stdout" + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "7OZElEvcGOD1", + "colab_type": "text" + }, + "source": [ + "\n", + "# Working with Joins" + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "UJBC7r3JFyCL", + "colab_type": "code", + "colab": {} + }, + "source": [ + "# Loading the data\n", + "from pyspark.sql.functions import col\n", + "df = spark.read.csv('reported-crimes.csv',header=True).withColumn('Date',to_timestamp(col('Date'),'MM/dd/yyyy hh:mm:ss a'))" + ], + "execution_count": 0, + "outputs": [] + }, + { + "cell_type": "code", + "metadata": { + "id": "-_aNpIqwIERe", + "colab_type": "code", + "outputId": "fc9c92ba-04e0-47e9-bf9b-1708e707334a", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 204 + } + }, + "source": [ + "# Dowloading police station data\n", + "!wget -O police-station.csv https://data.cityofchicago.org/api/views/z8bn-74gv/rows.csv?accessType=DOWNLOAD" + ], + "execution_count": 0, + "outputs": [ + { + "output_type": "stream", + "text": [ + "--2020-05-02 15:16:24-- https://data.cityofchicago.org/api/views/z8bn-74gv/rows.csv?accessType=DOWNLOAD\n", + "Resolving data.cityofchicago.org (data.cityofchicago.org)... 52.206.140.205, 52.206.140.199, 52.206.68.26\n", + "Connecting to data.cityofchicago.org (data.cityofchicago.org)|52.206.140.205|:443... connected.\n", + "HTTP request sent, awaiting response... 200 OK\n", + "Length: unspecified [text/csv]\n", + "Saving to: ‘police-station.csv’\n", + "\n", + "police-station.csv [ <=> ] 5.57K --.-KB/s in 0s \n", + "\n", + "2020-05-02 15:16:28 (587 MB/s) - ‘police-station.csv’ saved [5699]\n", + "\n" + ], + "name": "stdout" + } + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "HOQjNSrdIO1n", + "colab_type": "code", + "outputId": "d0bee83f-a994-4e86-f934-88f5364a3b8d", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 479 + } + }, + "source": [ + "ps = spark.read.csv(\"police-station.csv\", header=True)\n", + "ps.show()" + ], + "execution_count": 0, + "outputs": [ + { + "output_type": "stream", + "text": [ + "+--------------------+-----------------+--------------------+-------+-----+-----+--------------------+------------+------------+------------+------------+------------+-----------+------------+--------------------+\n", + "| DISTRICT| DISTRICT NAME| ADDRESS| CITY|STATE| ZIP| WEBSITE| PHONE| FAX| TTY|X COORDINATE|Y COORDINATE| LATITUDE| LONGITUDE| LOCATION|\n", + "+--------------------+-----------------+--------------------+-------+-----+-----+--------------------+------------+------------+------------+------------+------------+-----------+------------+--------------------+\n", + "| 1| Central| 1718 S State St|Chicago| IL|60616|http://home.chica...|312-745-4290|312-745-3694|312-745-3693| 1176569.052| 1891771.704|41.85837259|-87.62735617|(41.8583725929, -...|\n", + "| 6| Gresham| 7808 S Halsted St|Chicago| IL|60620|http://home.chica...|312-745-3617|312-745-3649|312-745-3639| 1172283.013| 1853022.646|41.75213684|-87.64422891|(41.7521368378, -...|\n", + "| 11| Harrison| 3151 W Harrison St|Chicago| IL|60612|http://home.chica...|312-746-8386|312-746-4281|312-746-5151| 1155244.069| 1897148.755|41.87358229|-87.70548813|(41.8735822883, -...|\n", + "| 16| Jefferson Park|5151 N Milwaukee Ave|Chicago| IL|60630|http://home.chica...|312-742-4480|312-742-4421|312-742-4423| 1138480.758| 1933660.473|41.97409445|-87.76614884|(41.9740944511, -...|\n", + "| Headquarters| Headquarters| 3510 S Michigan Ave|Chicago| IL|60653|http://home.chica...| null| null| null| 1177731.401| 1881697.404|41.83070169|-87.62339535|(41.8307016873, -...|\n", + "| 24| Rogers Park| 6464 N Clark St|Chicago| IL|60626|http://home.chica...|312-744-5907|312-744-6928|312-744-7603| 1164193.588| 1943199.401|41.99976348|-87.67132429|(41.9997634842, -...|\n", + "| 2| Wentworth|5101 S Wentworth Ave|Chicago| IL|60609|http://home.chica...|312-747-8366|312-747-5396|312-747-6656| 1175864.837| 1871153.753|41.80181109|-87.63056018|(41.8018110912, -...|\n", + "| 7| Englewood| 1438 W 63rd St|Chicago| IL|60636|http://home.chica...|312-747-8223|312-747-6558|312-747-6652| 1167659.235| 1863005.522|41.77963154|-87.66088702|(41.7796315359, -...|\n", + "| 25| Grand Central| 5555 W Grand Ave|Chicago| IL|60639|http://home.chica...|312-746-8605|312-746-4353|312-746-8383| 1138770.871| 1913442.439|41.91860889|-87.76557448|(41.9186088912, -...|\n", + "| 10| Ogden| 3315 W Ogden Ave|Chicago| IL|60623|http://home.chica...|312-747-7511|312-747-7429|312-747-7471| 1154500.753| 1890985.501|41.85668453|-87.70838196|(41.8566845327, -...|\n", + "| 15| Austin| 5701 W Madison St|Chicago| IL|60644|http://home.chica...|312-743-1440|312-743-1366|312-743-1485| 1138148.815| 1899399.078|41.88008346|-87.76819989|(41.8800834614, -...|\n", + "| 3| Grand Crossing|7040 S Cottage Gr...|Chicago| IL|60637|http://home.chica...|312-747-8201|312-747-5479|312-747-9168| 1182739.183| 1858317.732|41.76643089|-87.60574786|(41.7664308925, -...|\n", + "| 14| Shakespeare|2150 N California...|Chicago| IL|60647|http://home.chica...|312-744-8250|312-744-2422|312-744-8260| 1157304.426| 1914481.521|41.92110332|-87.69745182|(41.9211033246, -...|\n", + "| 8| Chicago Lawn| 3420 W 63rd St|Chicago| IL|60629|http://home.chica...|312-747-8730|312-747-8545|312-747-8116| 1154575.242| 1862672.049|41.77898719|-87.70886382|(41.778987189, -8...|\n", + "| 4| South Chicago| 2255 E 103rd St|Chicago| IL|60617|http://home.chica...|312-747-7581|312-747-5276|312-747-9169| 1193131.299| 1837090.265|41.70793329|-87.56834912|(41.7079332906, -...|\n", + "| 20| Lincoln| 5400 N Lincoln Ave|Chicago| IL|60625|http://home.chica...|312-742-8714|312-742-8803|312-742-8841| 1158399.146| 1935788.826|41.97954951|-87.69284451|(41.9795495131, -...|\n", + "| 18| Near North| 1160 N Larrabee St|Chicago| IL|60610|http://home.chica...|312-742-5870|312-742-5771|312-742-5773| 1172080.029| 1908086.527|41.90324165|-87.64335214|(41.9032416531, -...|\n", + "| 12| Near West|1412 S Blue Islan...| null| null| null| null| null| null| null| null| null| null| null| null|\n", + "|\",Chicago,IL,6060...| -87.6569725149)\"| null| null| null| null| null| null| null| null| null| null| null| null| null|\n", + "| 9| Deering| 3120 S Halsted St|Chicago| IL|60608|http://home.chica...|312-747-8227|312-747-5329|312-747-9172| 1171440.24| 1884085.224|41.83739443|-87.64640771|(41.8373944311, -...|\n", + "+--------------------+-----------------+--------------------+-------+-----+-----+--------------------+------------+------------+------------+------------+------------+-----------+------------+--------------------+\n", + "only showing top 20 rows\n", + "\n" + ], + "name": "stdout" + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "hY05yWIEJIdb", + "colab_type": "text" + }, + "source": [ + "**The reported crimes dataset has only the district number. Add the district name by joining with the police station dataset.**" + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "QePrc33RI25U", + "colab_type": "code", + "outputId": "41dd8527-bcdd-4a12-faa2-8cce3844b207", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 34 + } + }, + "source": [ + "# Caching the crimes dataset to speed things up, and then, since spark does lazy evaluation, gonna run an action to make it evaluated.\n", + "df.cache()\n", + "df.count()" + ], + "execution_count": 0, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "259013" + ] + }, + "metadata": { + "tags": [] + }, + "execution_count": 47 + } + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "lSsxX7qoJinQ", + "colab_type": "code", + "outputId": "0858b3ba-c6aa-4293-e807-66a789cc5eb3", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 901 + } + }, + "source": [ + "ps.select(col('DISTRICT')).distinct().show()\n", + "df.select(col('District')).distinct().show()" + ], + "execution_count": 0, + "outputs": [ + { + "output_type": "stream", + "text": [ + "+------------+\n", + "| DISTRICT|\n", + "+------------+\n", + "| 7|\n", + "| 15|\n", + "| 11|\n", + "| 3|\n", + "| 8|\n", + "| 22|\n", + "| 16|\n", + "| 5|\n", + "| 18|\n", + "| 17|\n", + "| 6|\n", + "| 19|\n", + "| 25|\n", + "|Headquarters|\n", + "| 24|\n", + "| 9|\n", + "| 1|\n", + "| 20|\n", + "| 10|\n", + "| 4|\n", + "+------------+\n", + "only showing top 20 rows\n", + "\n", + "+--------+\n", + "|District|\n", + "+--------+\n", + "| 009|\n", + "| 012|\n", + "| 024|\n", + "| 031|\n", + "| 015|\n", + "| 006|\n", + "| 019|\n", + "| 020|\n", + "| 011|\n", + "| 025|\n", + "| 003|\n", + "| 005|\n", + "| 016|\n", + "| 018|\n", + "| 008|\n", + "| 022|\n", + "| 001|\n", + "| 014|\n", + "| 010|\n", + "| 004|\n", + "+--------+\n", + "only showing top 20 rows\n", + "\n" + ], + "name": "stdout" + } + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "7keEhyBMJsFT", + "colab_type": "code", + "outputId": "bf93a549-5b5e-45d2-e939-3509a44573a8", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 459 + } + }, + "source": [ + "# Transfomring crime data to remove 0 from beginning, inroder to match the data\n", + "from pyspark.sql.functions import lpad\n", + "ps = ps.withColumn('Format_district',lpad(col('DISTRICT'),3,'0'))\n", + "ps.select(col('Format_district')).show()" + ], + "execution_count": 0, + "outputs": [ + { + "output_type": "stream", + "text": [ + "+---------------+\n", + "|Format_district|\n", + "+---------------+\n", + "| 001|\n", + "| 006|\n", + "| 011|\n", + "| 016|\n", + "| Hea|\n", + "| 024|\n", + "| 002|\n", + "| 007|\n", + "| 025|\n", + "| 010|\n", + "| 015|\n", + "| 003|\n", + "| 014|\n", + "| 008|\n", + "| 004|\n", + "| 020|\n", + "| 018|\n", + "| 012|\n", + "| \",C|\n", + "| 009|\n", + "+---------------+\n", + "only showing top 20 rows\n", + "\n" + ], + "name": "stdout" + } + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "U7Py4EYyKJTN", + "colab_type": "code", + "outputId": "c5da5c45-e024-46b9-a148-db4a7ce7da6a", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 479 + } + }, + "source": [ + "# Executing the join and deleting some column so that we don't get too much data\n", + "df.join(ps, df.District == ps.Format_district, 'left_outer').drop(\n", + " 'ADDRESS',\n", + " 'CITY',\n", + " 'STATE',\n", + " 'ZIP',\n", + " 'WEBSITE',\n", + " 'PHONE',\n", + " 'FAX',\n", + " 'TTY',\n", + " 'X COORDINATE',\n", + " 'Y COORDINATE',\n", + " 'LATITUDE',\n", + " 'LONGITUDE',\n", + " 'LOCATION').show(truncate=False)" + ], + "execution_count": 0, + "outputs": [ + { + "output_type": "stream", + "text": [ + "+--------+-----------+-------------------+-------------------------+----+--------------------------+-------------------------------------------------+--------------------------------------+------+--------+----+--------+----+--------------+--------+----+----------------------+--------+--------------+---------------+\n", + "|ID |Case Number|Date |Block |IUCR|Primary Type |Description |Location Description |Arrest|Domestic|Beat|District|Ward|Community Area|FBI Code|Year|Updated On |DISTRICT|DISTRICT NAME |Format_district|\n", + "+--------+-----------+-------------------+-------------------------+----+--------------------------+-------------------------------------------------+--------------------------------------+------+--------+----+--------+----+--------------+--------+----+----------------------+--------+--------------+---------------+\n", + "|12040452|JD220630 |2019-11-21 12:00:00|004XX N WABASH AVE |1153|DECEPTIVE PRACTICE |FINANCIAL IDENTITY THEFT OVER $ 300 |RESIDENCE |false |false |1834|018 |42 |8 |11 |2019|05/01/2020 03:49:55 PM|18 |Near North |018 |\n", + "|12039199|JD219081 |2019-04-01 08:00:00|021XX S INDIANA AVE |1153|DECEPTIVE PRACTICE |FINANCIAL IDENTITY THEFT OVER $ 300 |RESIDENCE |false |false |0132|001 |3 |33 |11 |2019|05/01/2020 03:48:05 PM|1 |Central |001 |\n", + "|12040466|JD220650 |2019-11-07 12:00:00|026XX N NEW ENGLAND AVE |1152|DECEPTIVE PRACTICE |ILLEGAL USE CASH CARD |RESIDENCE |false |false |2512|025 |36 |18 |11 |2019|05/01/2020 03:49:55 PM|25 |Grand Central |025 |\n", + "|11938499|JD100565 |2019-12-31 11:30:00|070XX S COTTAGE GROVE AVE|0460|BATTERY |SIMPLE |OTHER (SPECIFY) |true |false |0321|003 |6 |42 |08B |2019|05/01/2020 03:48:05 PM|3 |Grand Crossing|003 |\n", + "|11864640|JC477069 |2019-10-18 09:56:00|002XX S LAVERGNE AVE |041A|BATTERY |AGGRAVATED - HANDGUN |SIDEWALK |false |false |1533|015 |28 |25 |04B |2019|05/01/2020 03:48:05 PM|15 |Austin |015 |\n", + "|11766889|JC359918 |2019-07-17 15:07:00|024XX W FOSTER AVE |0610|BURGLARY |FORCIBLE ENTRY |RESIDENCE - PORCH / HALLWAY |true |false |2011|020 |40 |4 |05 |2019|05/01/2020 03:48:05 PM|20 |Lincoln |020 |\n", + "|11740807|JC328262 |2019-06-30 05:04:00|048XX W AUGUSTA BLVD |041A|BATTERY |AGGRAVATED - HANDGUN |SIDEWALK |false |false |1531|015 |37 |25 |04B |2019|05/01/2020 03:48:05 PM|15 |Austin |015 |\n", + "|11718084|JC300874 |2019-05-13 13:00:00|062XX S EMERALD DR |1750|OFFENSE INVOLVING CHILDREN|CHILD ABUSE |GOVERNMENT BUILDING / PROPERTY |false |true |0711|007 |16 |68 |08B |2019|05/01/2020 03:48:05 PM|7 |Englewood |007 |\n", + "|11647696|JC215453 |2019-04-07 17:56:00|062XX S FRANCISCO AVE |041A|BATTERY |AGGRAVATED - HANDGUN |ALLEY |false |false |0823|008 |16 |66 |04B |2019|05/01/2020 03:48:05 PM|8 |Chicago Lawn |008 |\n", + "|12039931|JD220113 |2019-10-18 08:00:00|120XX S LAFLIN ST |2826|OTHER OFFENSE |HARASSMENT BY ELECTRONIC MEANS |RESIDENCE |false |false |0524|005 |34 |53 |26 |2019|04/30/2020 03:50:06 PM|5 |Calumet |005 |\n", + "|12039974|JD220130 |2019-12-24 23:00:00|049XX N ALBANY AVE |0890|THEFT |FROM BUILDING |RESIDENCE |false |false |1713|017 |33 |14 |06 |2019|04/30/2020 03:50:06 PM|17 |Albany Park |017 |\n", + "|12039885|JD219909 |2019-10-01 02:00:00|021XX S HARDING AVE |1752|OFFENSE INVOLVING CHILDREN|AGGRAVATED CRIMINAL SEXUAL ABUSE BY FAMILY MEMBER|RESIDENCE |false |true |1014|010 |24 |29 |17 |2019|04/30/2020 03:50:06 PM|10 |Ogden |010 |\n", + "|12039884|JD220045 |2019-07-01 12:00:00|026XX W BALMORAL AVE |1153|DECEPTIVE PRACTICE |FINANCIAL IDENTITY THEFT OVER $ 300 |RESIDENCE |false |false |2011|020 |40 |4 |11 |2019|04/30/2020 03:50:06 PM|20 |Lincoln |020 |\n", + "|12038367|JD218291 |2019-04-21 12:00:00|071XX S CYRIL AVE |0890|THEFT |FROM BUILDING |APARTMENT |false |false |0333|003 |5 |43 |06 |2019|04/30/2020 03:47:59 PM|3 |Grand Crossing|003 |\n", + "|12038544|JD218327 |2019-12-22 13:00:00|028XX S CHRISTIANA AVE |5002|OTHER OFFENSE |OTHER VEHICLE OFFENSE |STREET |false |false |1032|010 |22 |30 |26 |2019|04/30/2020 03:47:59 PM|10 |Ogden |010 |\n", + "|12038308|JD218196 |2019-07-22 08:00:00|052XX S ARCHER AVE |0910|MOTOR VEHICLE THEFT |AUTOMOBILE |PARKING LOT / GARAGE (NON RESIDENTIAL)|false |false |0815|008 |23 |57 |07 |2019|04/30/2020 03:47:59 PM|8 |Chicago Lawn |008 |\n", + "|12040067|JD220150 |2019-12-15 18:00:00|057XX W ROOSEVELT RD |0265|CRIMINAL SEXUAL ASSAULT |AGGRAVATED - OTHER |HOSPITAL BUILDING / GROUNDS |false |false |1513|015 |29 |25 |02 |2019|04/30/2020 03:50:06 PM|15 |Austin |015 |\n", + "|12039858|JD219964 |2019-09-03 09:00:00|003XX S SPRINGFIELD AVE |1154|DECEPTIVE PRACTICE |FINANCIAL IDENTITY THEFT $300 AND UNDER |APARTMENT |false |false |1133|011 |28 |26 |11 |2019|04/30/2020 03:50:06 PM|11 |Harrison |011 |\n", + "|12038342|JD218257 |2019-05-01 00:00:00|002XX N MENARD AVE |1154|DECEPTIVE PRACTICE |FINANCIAL IDENTITY THEFT $300 AND UNDER |RESIDENCE |false |false |1512|015 |29 |25 |11 |2019|04/30/2020 03:47:59 PM|15 |Austin |015 |\n", + "|12038644|JD218194 |2019-06-01 13:45:00|003XX N MENARD AVE |1153|DECEPTIVE PRACTICE |FINANCIAL IDENTITY THEFT OVER $ 300 |null |false |false |1512|015 |29 |25 |11 |2019|04/30/2020 03:47:59 PM|15 |Austin |015 |\n", + "+--------+-----------+-------------------+-------------------------+----+--------------------------+-------------------------------------------------+--------------------------------------+------+--------+----+--------+----+--------------+--------+----+----------------------+--------+--------------+---------------+\n", + "only showing top 20 rows\n", + "\n" + ], + "name": "stdout" + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "vj0mPaHU5i5n", + "colab_type": "text" + }, + "source": [ + "As you can see, we have done a left outer join between two dataframes. The following joins are supported by PySpark:\n", + "1. inner (default)\n", + "2. cross\n", + "3. outer\n", + "4. full\n", + "5. full_outer\n", + "6. left\n", + "7. left_outer\n", + "8. right\n", + "9. right_outer\n", + "10. left_semi\n", + "11. left_anti" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "EEEB2TVqL4Ie", + "colab_type": "text" + }, + "source": [ + "\n", + "# Hands-on again!" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "50gdkX-_MEMv", + "colab_type": "text" + }, + "source": [ + "**What is the most frequently reported non-criminal activity?**" + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "pbjuihaOL-fM", + "colab_type": "code", + "colab": {} + }, + "source": [ + "" + ], + "execution_count": 0, + "outputs": [] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "UdMXUsjKME2y", + "colab_type": "text" + }, + "source": [ + "**Find the day of the week with the most reported crime?**" + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "7Sdq1ZwRMN0p", + "colab_type": "code", + "colab": {} + }, + "source": [ + "" + ], + "execution_count": 0, + "outputs": [] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "REpHzg_nNctg", + "colab_type": "text" + }, + "source": [ + "**Using a bar chart, plot which day of the week has the most number of reported crime.**" + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "j5olv2qkNfPa", + "colab_type": "code", + "outputId": "60bc50d8-9213-425d-da1b-5b73b2aa839a", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 51 + } + }, + "source": [ + "from pyspark.sql.functions import date_format, col\n", + "dow = [x[0] for x in df.groupBy(date_format(col('Date'),'E')).count().collect()]\n", + "print(dow)\n", + "cnt = [x[1] for x in df.groupBy(date_format(col('Date'),'E')).count().collect()]\n", + "print(cnt)" + ], + "execution_count": 0, + "outputs": [ + { + "output_type": "stream", + "text": [ + "['Sun', 'Mon', 'Thu', 'Sat', 'Wed', 'Tue', 'Fri']\n", + "[36026, 36930, 36235, 37978, 36095, 36855, 38894]\n" + ], + "name": "stdout" + } + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "gzOCDOqFOOc6", + "colab_type": "code", + "outputId": "8cb2e216-8272-4036-dd7b-ddf3b4abcdc8", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 204 + } + }, + "source": [ + "import pandas as pd\n", + "import matplotlib.pyplot as plt\n", + "\n", + "cp = pd.DataFrame({'Day_of_week':dow, 'Count':cnt})\n", + "cp.head()" + ], + "execution_count": 0, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/html": [ + "
      \n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
      Day_of_weekCount
      0Sun36026
      1Mon36930
      2Thu36235
      3Sat37978
      4Wed36095
      \n", + "
      " + ], + "text/plain": [ + " Day_of_week Count\n", + "0 Sun 36026\n", + "1 Mon 36930\n", + "2 Thu 36235\n", + "3 Sat 37978\n", + "4 Wed 36095" + ] + }, + "metadata": { + "tags": [] + }, + "execution_count": 52 + } + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "J4pxrZeSOvIS", + "colab_type": "code", + "outputId": "bd39d130-024a-4fa5-d013-aefdade2e701", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 324 + } + }, + "source": [ + "cp.sort_values('Count', ascending=False).plot(kind='bar', color= 'red', x='Day_of_week', y='Count')\n", + "plt.xlabel(\"Day of week\")\n", + "plt.ylabel(\"Number of reported crimes\")\n", + "plt.title(\"No.of reported crimes per day\")" + ], + "execution_count": 0, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "Text(0.5, 1.0, 'No.of reported crimes per day')" + ] + }, + "metadata": { + "tags": [] + }, + "execution_count": 53 + }, + { + "output_type": "display_data", + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZEAAAEiCAYAAAA4f++MAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAgAElEQVR4nO3deZyVZf3/8ddbQCFRcSFSUEHFyuUr6rhkZmWpWCpmVtoimYmWln5LU9sorczKJVNTXBJbRH8uSaYRmbgvDIoLLl8nRRlyQVABTQz8/P64rpHjOHPmnsOcOXOY9/PxOI9z7uvePucwnM+5r+u+rksRgZmZWSVWqXUAZmZWv5xEzMysYk4iZmZWMScRMzOrmJOImZlVzEnEzMwq5iRiPY6kD0p6QtJiSfvXOp6uJulHkv7QRce6UdLYrjhWvZA0W9LHax2HJU4iVkj+j/uCpNVLyr4qaVoVTncycE5EDIyIP1fh+CukJ32JRcTeETGx1nFY7+UkYp3RBzimG86zMTCryIaS+lY5lpqcqyNKVqr/vz3p87XiVqo/Qqu6XwLHSRrU1kpJu0iaLumV/LxLeweSdLikJkkLJE2WtEEu/xewCfCXXJ21Whv7zpZ0gqQHgVcl9ZW0s6Q7Jb0s6QFJHynZfpqkUyXdK2mhpOskrVOyfj9Js/K+0yS9v8y5Lgc2KonvO3m7cucfIekWSYskTQXWK/chSxojaWaO9V+SRpe8j59KugN4Ddgkl301r/+ypDsknZnjeDL/m3xZ0px8JTm25DyrSfqVpGckPS/pfEkD8rr1JF2fj7NA0m3tJS1JIemb+XwvSvpl6baSviLpUUkvSZoiaeNW+x4l6QngiXaO/yVJT0uaL+l7rdbtKOmuHOezks6RtGped66k01ttP1nS/5b7/K2TIsIPPzp8ALOBjwPXAD/JZV8FpuXX6wAvAV8C+gIH5+V12zjW7sCLwHbAasBvgFtbn6uDWGYCGwIDgKHAfOATpB9Ge+TlwXn7acBcYCtgdeBq4A953ebAq3mffsB3gCZg1bbO1VZ8Bc5/F3BGfq+7AYtazt/Ge9sReCUfY5V87PeVvI9ngC3zZ9wvl301r/8ysBQ4lHTV+JO8/bn53Hvmcw/M258JTM7/dmsAfwFOzetOBc7P5+gHfAhQOzEHcHM+zkbA/5XENCZ/nu/PMX8fuLPVvlPzvgPaOPYWwOL8ua2WP8elLZ8/sD2wcz72cOBR4NiSz/LfwCp5eT1S8h1S6/9PK9Oj5gH4UR8PlieRrfKX3GDenkS+BNzbap+7gC+3cayLgV+ULA8E/gsMLz1XB7F8pWT5BOD3rbaZAozNr6cBPy9ZtwXwRv6i/QFwZcm6VUgJ5yNtnaut+MqdP3+pLgVWL1n3J9pPIhcAZ7azbhpwchtlpUnkiZJ1W+cv6SElZfOBUYBIyXPTknUfAJ7Kr08GrgM2K/C3EcDokuWvAzfl1zcCh7X6fF8DNi7Zd/cyx/4hMKlkefX8b9fm3wdwLHBtyfKjwB759dHADbX+v7SyPVydZZ0SEQ8D1wMntlq1AfB0q7KnSb+kW3vbthGxmPTl1ta27ZlT8npj4DO5SuNlSS8DuwLrt7P906Rf1+u1Ecubeduh7ezblnLn3wB4KSJebXX+9mwI/KvM+o5ieb7k9X8AIqJ12UDSj4B3ATNKYv5bLodUddkE/D1XU7X+9y4X19Ok9w3ps/l1yTkWkBJY0c93g9L1+XOc37IsafNc7facpIXAz3h7deFE4Iv59ReB33fwPqyTnESsEuOBw3n7F8G/SV8YpTYi/apv7W3bKt3xtW4727andPjpOaQrgUElj9Uj4ucl22zYKq7/kqrUWseivG1pLK2Hum69XO78zwJrq+Sutnz+9swBNi2zvquG3X6RlFC2LIl5rYgYCBARiyLi2xGxCbAf8C1JHytzvNaf77/z6znAEa0+mwERcWfB9/Rs6bElvYv0t9Lit8BjwMiIWBP4LilJtfgDMEbSNqQqtR53t1+9cxKxTouIJuAK4JslxTcAm0v6fG7o/hyp2uj6Ng5xOXCopFG54fxnwD0RMbvCkP4A7CtpL0l9JPWX9BFJw0q2+aKkLfKX0MnAVRGxDLgS+KSkj0nqB3wbWALc+Y6zLPc8qfG/w/NHxNNAI/BjSatK2hXYt8yxLyZ9Nh+TtIqkoZLe19kPpCP5iutC4ExJ7wbI59orv95H0mY5qb4CLAPeLHPI4yWtLWlD0h18V+Ty84GTJG2Zj7uWpM90ItSrgH0k7ZobzE/m7d9bawALgcX5c/paq/fZDEwnXYFcHRH/6cS5rQAnEavUyaT6aQAiYj6wD+lLeD6pgXqfiHgRQOnupy/kbf9Baou4mvRLc1PgoEoDiYg5pAbc7wLzSL9+j+ftf9+/By4FngP6kxNgRDxOqub4DenX+b7AvhHxRplTngp8P1fRHFfg/J8HdiJV5YwHLivzXu4lNYyfSfryvoV3XuF1lRNIVVZ356qgfwDvzetG5uXFpLat8yLi5jLHug6YQboJ4a+kZEhEXAucBkzK53gY2LtogBExCziK1I70LOlmjeaSTY4jfb6LSEnxitbHIFVpbY2rsqpCEZ6UylZuSh0i/xARF9U6lpWRpCBVJzXVOpa2SNqNdLW4cfgLr8v5SsTMVlq5ivIY4CInkOpwEjGzlZJSp9GXSXfJnVXjcFZars4yM7OK+UrEzMwq5iRiZmYV63WjZq633noxfPjwWodhZlZXZsyY8WJEDG5dXvUkIqkPqbPV3IjYR9IIYBKp1+kM4EsR8UbudHYZaUC1+cDnWjqfSToJOIzU4embETEll48Gfk0aA+miVj2U2zR8+HAaGxu7+F2ama3cJLU5XE93VGcdQxoErcVppAHmNiN1HDoslx9GGmNoM1JHq9MAJG1B6oi2JTAaOC/3Cu5DGp10b1LP6IPztmZm1k2qmkTysBOfBC7KyyINA35V3mQi0DL96Zi8TF7/sbz9GNIonksi4ilSD9sd86MpIp7MvYsn5W3NzKybVPtK5CzS8BctY+6sC7wcEUvzcjPLB/EbSh6tM69/JW//VnmrfdorfwdJ4yQ1SmqcN2/eir4nMzPLqtYmImkf4IWImKGSWd5qISImABMAGhoa3DHGzNr03//+l+bmZl5//fVah1Iz/fv3Z9iwYfTr16/Q9tVsWP8gsJ+kT5AGvFuT1Ag+SFLffLUxjOVDbs8lDfncrDTX8lqkBvaW8hal+7RXbmbWac3NzayxxhoMHz6cVJveu0QE8+fPp7m5mREjRhTap2rVWRFxUkQMi4jhpIbxf0bEF0jTaB6YNxtLGv0T0jSdLfM/H5i3j1x+kNJ80CNIo4veSxreeaTS/NWr5nNMrtb7MbOV3+uvv866667bKxMIgCTWXXfdTl2J1aKfyAmkYaF/AtxPHjI6P/9eUhNpyOyDIA0FLelK4BHSNKNH5XkgkHQ0aRrSPsAledhoM7OK9dYE0qKz779bkkhETCPNBU1EPEm6s6r1Nq8DbU5WExE/BX7aRvkNpMmQzMxWCs899xzHHnss06dPZ9CgQQwZMoSzzjqLzTffvEuOP23aNFZddVV22WWXLjler+ux3mnV/lXiATDNeq6u/v/fwf/3iOBTn/oUY8eOZdKkSQA88MADPP/8812aRAYOHNhlScRjZ5mZ9RA333wz/fr148gjj3yrbJtttmHXXXfl+OOPZ6uttmLrrbfmiivSBI7Tpk1jn332eWvbo48+mksvvRRIo3OMHz+e7bbbjq233prHHnuM2bNnc/7553PmmWcyatQobrvtthWO2VciZmY9xMMPP8z222//jvJrrrmGmTNn8sADD/Diiy+yww47sNtuu3V4vPXWW4/77ruP8847j1/96ldcdNFFHHnkkQwcOJDjjjuuS2L2lYiZWQ93++23c/DBB9OnTx+GDBnChz/8YaZPn97hfgcccAAA22+/PbNnz65KbE4iZmY9xJZbbsmMGTMKb9+3b1/efPPNt5Zb35q72mqrAdCnTx+WLl1KNTiJrMyk6j7MrEvtvvvuLFmyhAkTJrxV9uCDDzJo0CCuuOIKli1bxrx587j11lvZcccd2XjjjXnkkUdYsmQJL7/8MjfddFOH51hjjTVYtGhRl8XsNhEzsx5CEtdeey3HHnssp512Gv3792f48OGcddZZLF68mG222QZJ/OIXv+A973kPAJ/97GfZaqutGDFiBNtuu22H59h333058MADue666/jNb37Dhz70oRWLubfNsd7Q0BCdmk+knm/xrefYzWrg0Ucf5f3vf3+tw6i5tj4HSTMioqH1tq7OMjOzijmJmJlZxdwmYj2Xq+PMejwnEbNqcRKsSxHRqwdh7Gw7uauzzMyy/v37M3/+/E5/ka4sWuYT6d+/f+F9fCViZpYNGzaM5uZmevM02i0zGxblJGJmbeuF1XH9+vUrPKOfJU4iZrZyqvckWCfxu03EzMwqVrUkIqm/pHslPSBplqQf5/JLJT0laWZ+jMrlknS2pCZJD0raruRYYyU9kR9jS8q3l/RQ3uds9eZbKszMaqCa1VlLgN0jYrGkfsDtkm7M646PiKtabb83MDI/dgJ+C+wkaR1gPNAABDBD0uSIeClvczhwD2ma3NHAjZiZWbeo2pVIJIvzYr/8KFcJNwa4LO93NzBI0vrAXsDUiFiQE8dUYHRet2ZE3B3pfrzLgP2r9X7MzOydqtomIqmPpJnAC6REcE9e9dNcZXWmpNVy2VBgTsnuzbmsXHlzG+VmZtZNqppEImJZRIwChgE7StoKOAl4H7ADsA5wQjVjAJA0TlKjpMbefP+3mVlX65a7syLiZeBmYHREPJurrJYAvwN2zJvNBTYs2W1YLitXPqyN8rbOPyEiGiKiYfDgwV3xlszMjOrenTVY0qD8egCwB/BYbssg30m1P/Bw3mUycEi+S2tn4JWIeBaYAuwpaW1JawN7AlPyuoWSds7HOgS4rlrvx8zM3qmad2etD0yU1IeUrK6MiOsl/VPSYEDATODIvP0NwCeAJuA14FCAiFgg6RSgZVb6kyNiQX79deBSYADprizfmWVm1o08s2FH6qTXaJvqOXZw/B1x/OU5/vI6Gb9nNjQzsy7nJGJmZhVzEjEzs4o5iZiZWcWcRMzMrGJOImZmVjEnETMzq1iHSUTSZyStkV9/X9I1pXN9mJlZ71XkSuQHEbFI0q7Ax4GLSfN4mJlZL1ckiSzLz58EJkTEX4FVqxeSmZnViyJJZK6kC4DPATfk+T/clmJmZoWSwWdJI+nulYd0Xwc4vqpRmZlZXegwiUTEa6SZCXfNRUuBJ6oZlJmZ1Ycid2eNJ80+eFIu6gf8oZpBmZlZfShSnfUpYD/gVYCI+DewRjWDMjOz+lAkibwRadKRAJC0enVDMjOzelEkiVyZ784aJOlw4B/AhdUNy8zM6kGH0+NGxK8k7QEsBN4L/DAiplY9MjMz6/EK9ffISeMU4GfADEnrdLSPpP6S7pX0gKRZkn6cy0dIukdSk6QrJK2ay1fLy015/fCSY52Uyx+XtFdJ+ehc1iTpxE69czMzW2FF7s46QtJzwINAIzAjP3dkCbB7RGwDjAJGS9oZOA04MyI2A14CDsvbHwa8lMvPzNshaQvgIGBLYDRwnqQ+kvoA5wJ7A1sAB+dtzcysmxS5EjkO2CoihkfEJhExIiI26WinSBbnxX75EcDuwFW5fCKwf349Ji+T139MknL5pIhYEhFPAU3AjvnRFBFPRsQbwKS8rZmZdZMiSeRfwGuVHDxfMcwkdVacmo/1ckQszZs0A0Pz66HAHIC8/hVg3dLyVvu0V95WHOMkNUpqnDdvXiVvxczM2tBhwzqpk+Gdku4hVVEBEBHf7GjHiFgGjJI0CLgWeF+lga6IiJgATABoaGiIWsRgZrYyKpJELgD+CTwEvFnJSSLiZUk3Ax8g3SrcN19tDAPm5s3mAhsCzZL6AmsB80vKW5Tu0165mZl1gyJJpF9EfKuzB5Y0GPhvTiADgD1IjeU3AweS2jDGAtflXSbn5bvy+n9GREiaDPxJ0hnABsBI4F5AwEhJI0jJ4yDg852N08zMKlckidwoaRzwF95enbWgg/3WBybmu6hWAa6MiOslPQJMkvQT4H7SJFfk599LagIWkJICETFL0pXAI6TBH4/K1WRIOpo0wnAf4JKImFXkTZuZWddQGtGkzAbSU20UR5E7tHqihoaGaGwscodyJlUvGIAOPv8VUs+xg+PviOMvz/GX18n4Jc2IiIbW5UV6rI/o1JnMzKzXaDeJSNo9Iv4p6YC21kfENdULy8zM6kG5K5EPk+7K2reNdQE4iZiZ9XLtJpGIGC9pFeDGiLiyG2MyM7M6UbbHekS8CXynm2IxM7M6U2TYk39IOk7ShpLWaXlUPTIzM+vxivQT+Vx+PqqkLIC6vMXXzMy6jm/xNTOzihWZT+SoPIBiy/Lakr5e3bDMzKweFGkTOTwiXm5ZiIiXgMOrF5KZmdWLIkmkT54cCkhzhACrVi8kMzOrF0Ua1v8GXCHpgrx8RC4zM7NerkgSOQEYB3wtL08FLqpaRGZmVjeK3J31JnB+fpiZmb2lSJuImZlZm5xEzMysYk4iZmZWsXLzifyFNLxJmyJiv6pEZGZmdaPclcivgNOBp4D/ABfmx2LgXx0dOA/YeLOkRyTNknRMLv+RpLmSZubHJ0r2OUlSk6THJe1VUj46lzVJOrGkfISke3L5FZLcf8XMrBuVm0/kFgBJp7eaV/cvkopMUr4U+HZE3CdpDWCGpKl53ZkR8avSjSVtARwEbAlsQBo9ePO8+lxgD6AZmC5pckQ8ApyWjzVJ0vnAYcBvC8RmZmZdoEibyOqS3hqxV9IIYPWOdoqIZyPivvx6EfAoMLTMLmOASRGxJCKeApqAHfOjKSKejIg3gEnAmNyLfnfgqrz/RGD/Au/HzMy6SJEk8r/ANEnTJN0C3Awc25mTSBoObAvck4uOlvSgpEskrZ3LhgJzSnZrzmXtla8LvBwRS1uVt3X+cZIaJTXOmzevM6GbmVkZHSaRiPgbMBI4Bvgm8N6ImFL0BJIGAlcDx0bEQlJ106bAKOBZUrtLVUXEhIhoiIiGwYMHV/t0Zma9RpGh4N8FHA8cHREPABtJ2qfIwSX1IyWQP0bENQAR8XxELMs94S8kVVcBzAU2LNl9WC5rr3w+MEhS31blZmbWTYpUZ/0OeAP4QF6eC/yko51ym8XFwKMRcUZJ+folm30KeDi/ngwcJGm13O4yErgXmA6MzHdirUpqfJ8cEUGqWjsw7z8WuK7A+zEzsy5SZADGTSPic5IOBoiI10qHhi/jg8CXgIckzcxl3wUOljSK1AdlNmlUYCJilqQrgUdId3YdFRHLACQdDUwB+gCXRMSsfLwTgEmSfgLcT0paZmbWTYokkTckDSB3PJS0KbCko50i4nagrWRzQ5l9fgr8tI3yG9raLyKeZHl1mJmZdbMiSeRHpPlDNpT0R9IVxqHVDMrMzOpDkaHg/y5pBrAz6crimIh4seqRmZlZj1fk7qybImJ+RPw1Iq6PiBcl3dQdwZmZWc9WbgDG/sC7gPVyh8CW9o01Kd/z3MzMeoly1VlHkHqmbwDMYHkSWQicU+W4zMysDpQbgPHXks4BvhsRp3RjTGZmVifKtonkfhoHdFMsZmZWZ4r0WL9J0qcLdjA0M7NepEgSOQL4f6ROhwslLZK0sMpxmZlZHSjST2SN7gjEzMzqT5Ee60jaD9gtL06LiOurF5KZmdWLIp0Nf06aS+SR/DhG0qnVDszMzHq+IlcinwBG5fk/kDSRNGLuSdUMzMzMer4iDesAg0per1WNQMzMrP4UuRI5Fbhf0s2kXuu7ASdWNSozM6sLRe7OulzSNGAH0pwiJ0TEc9UOzMzMer5Cd2eRpsbdlZRE+gLXVi0iMzOrG0XuzjoPOBJ4iDQf+hGSzi2w34aSbpb0iKRZko7J5etImirpify8di6XpLMlNUl6UNJ2Jccam7d/QtLYkvLtJT2U9znbverNzLpXkYb13YG9IuJ3EfE70t1auxfYbynw7YjYgjSh1VGStiC1p9wUESOBm1jevrI3MDI/xgG/hZR0gPHATqSpcMe3JJ68zeEl+40uEJeZmXWRIkmkCdioZHnDXFZWRDwbEffl14uAR0nzkIwBJubNJgL759djgMsiuRsYJGl9YC9gakQsiIiXgKnA6LxuzYi4OyICuKzkWGZm1g2KtImsATwq6V5Sm8iOQKOkyQARsV9HB5A0HNgWuAcYEhHP5lXPAUPy66HAnJLdmnNZufLmNsrNzKybFEkiP1yRE0gaCFwNHBsRC0ubLSIiJMWKHL9gDONIVWRstNFGHWxtZmZFdVidFRG3ALOBfvn1vcB9EXFLXm6XpH6kBPLHiLgmFz+fq6LIzy/k8rmkqrIWw3JZufJhbZS39R4mRERDRDQMHjy4g3dsZmZFFbk763DgKuCCXDQM+HOB/QRcDDwaEWeUrJoMtNxhNRa4rqT8kHyX1s7AK7naawqwp6S1c4P6nsCUvG6hpJ3zuQ4pOZaZmXWDItVZR5HaQe4BiIgnJL27wH4fBL4EPCRpZi77LvBz4EpJhwFPA5/N624g3fnVBLwGHJrPt0DSKcD0vN3JEbEgv/46cCkwALgxP8zMrJsUSSJLIuKNlrYMSX1JDexlRcTtpGFS2vKxNrYPUsJq61iXAJe0Ud4IbNVRLGZmVh1FbvG9RdJ3gQGS9iDNcviX6oZlZmb1oEgSOQGYR+qxfgSp2un71QzKzMzqQ9nqLEl9gFkR8T7gwu4JyczM6kXZK5GIWAY8LsmdK8zM7B2KNKyvDczKPdZfbSks0lPdzMxWbkWSyA+qHoWZmdWlIpNSle2VbmZmvVfROdbNzMzewUnEzMwq1m4SkXRTfj6t+8IxM7N6Uq5NZH1JuwD7SZpEqyFMWiacMjOz3qtcEvkh6c6sYcAZrdYFxabINTOzlVi7SSQirgKukvSDiDilG2MyM7M6UeQW31Mk7QfsloumRcT11Q3LzMzqQZFJqU4FjgEeyY9jJP2s2oGZmVnPV6TH+ieBURHxJoCkicD9pAmmzMysFyvaT2RQyeu1qhGImZnVnyJXIqcC90u6mXSb727AiVWNyszM6kKHVyIRcTmwM3ANcDXwgYi4oqP9JF0i6QVJD5eU/UjSXEkz8+MTJetOktQk6XFJe5WUj85lTZJOLCkfIemeXH6FpFWLv20zM+sKhaqzIuLZiJicH88VPPalwOg2ys+MiFH5cQOApC2Ag4At8z7nSeqTJ8U6F9gb2AI4OG8LcFo+1mbAS8BhBeMyM7MuUrWxsyLiVmBBwc3HAJMiYklEPAU0ATvmR1NEPBkRbwCTgDGSROrseFXefyKwf5e+ATMz61AtBmA8WtKDubpr7Vw2FJhTsk1zLmuvfF3g5YhY2qq8TZLGSWqU1Dhv3ryueh9mZr1e2SSSq5Qe68Lz/RbYFBgFPAuc3oXHbldETIiIhohoGDx4cHec0sysV+jWOdYj4vmIWJb7nFxIqq4CmAtsWLLpsFzWXvl8YJCkvq3KzcysGxWpzmqZY/0mSZNbHpWcTNL6JYufAlru3JoMHCRpNUkjgJHAvcB0YGS+E2tVUuP75IgI4GbgwLz/WOC6SmIyM7PKVW2OdUmXAx8B1pPUDIwHPiJpFGkU4NnAEQARMUvSlaRhVZYCR+WrICQdDUwB+gCXRMSsfIoTgEmSfkLqQX9xJXGamVnllH7Ud7CRtDEwMiL+IeldQJ+IWFT16KqgoaEhGhsbi+8gdbzNiijw+VesnmMHx98Rx1+e4y+vk/FLmhERDa3LiwzAeDjpVtoLctFQ4M+dOruZma2UirSJHAV8EFgIEBFPAO+uZlBmZlYfiiSRJbmjHwD5jqgqX8eZmVk9KJJEbpH0XWCApD2A/wf8pbphmZlZPSiSRE4E5gEPke6mugH4fjWDMjOz+lBketw380RU95CqsR6PIrd0mZnZSq/DJCLpk8D5wL9I84mMkHRERNxY7eDMzKxnK9LZ8HTgoxHRBCBpU+CvgJOImVkvV6RNZFFLAsmeBOqyo6GZmXWtdq9EJB2QXzZKugG4ktQm8hnSmFZmZtbLlavO2rfk9fPAh/PrecCAqkVkZmZ1o90kEhGHdmcgZmZWf4rcnTUC+AYwvHT7iNivemGZmVk9KHJ31p9Jw6z/BXizuuGYmVk9KZJEXo+Is6seiZmZ1Z0iSeTXksYDfweWtBRGxH1Vi8rMzOpCkSSyNfAlYHeWV2dFXjYzs16sSGfDzwCbRMSHI+Kj+dFhApF0iaQXJD1cUraOpKmSnsjPa+dySTpbUpOkByVtV7LP2Lz9E5LGlpRvL+mhvM/ZUrWnATMzs9aKJJGHgUEVHPtSYHSrshOBmyJiJHBTXgbYGxiZH+OA30JKOqS52XcCdgTGtySevM3hJfu1PpeZmVVZkSQyCHhM0hRJk1seHe0UEbcCC1oVjwEm5tcTgf1Lyi+L5G5gkKT1gb2AqRGxICJeAqYCo/O6NSPi7jyi8GUlxzIzs25SpE1kfBeeb0hEPJtfPwcMya+HAnNKtmvOZeXKm9soNzOzblRkPpFbqnHiiAhJ3TIviaRxpGoyNtpoo+44pZlZr9BhdZakRZIW5sfrkpZJWljh+Z7PVVHk5xdy+Vxgw5LthuWycuXD2ihvU0RMiIiGiGgYPHhwhaGbmVlrHSaRiFgjItaMiDVJAy9+GjivwvNNBlrusBoLXFdSfki+S2tn4JVc7TUF2FPS2rlBfU9gSl63UNLO+a6sQ0qOZWZm3aRIw/pbcsP3n0kN3mVJuhy4C3ivpGZJhwE/B/aQ9ATw8bwMad72J4Em4ELg6/l8C4BTSEPPTwdOzmXkbS7K+/wLT5JlZtbt1NF06SXzikBKOg3AhyPiA9UMrFoaGhqisbGx+A7V7n5Szenq6zl2cPwdcfzlOf7yOhm/pBkR0dC6vMjdWaXziiwFZpNuyTUzs16uyN1ZnlfEzMzaVG563B+W2S8i4pQqxGNmZnWk3JXIq22UrQ4cBqxLavA2M7NerNz0uKe3vJa0BnAMcCgwCTi9vf3MzKz3KNsmkgdA/BbwBfXqaBAAAAzgSURBVNJYV9vlMazMzMzKton8EjgAmABsHRGLuy0qMzOrC+U6G34b2AD4PvDvkqFPFq3AsCdmZrYSKdcm0qne7GZm1vs4UZiZWcWcRMzMrGJOImZmVjEnETMzq5iTiJmZVcxJxMzMKuYkYmZmFXMSMTOzijmJmJlZxWqSRCTNlvSQpJmSGnPZOpKmSnoiP6+dyyXpbElNkh6UtF3Jccbm7Z+QNLYW78XMrDer5ZXIRyNiVMmcvScCN0XESOCmvAywNzAyP8YBv4W3RhgeD+wE7AiMb0k8ZmbWPXpSddYY0nDz5Of9S8ovi+RuYJCk9YG9gKkRsSAPTz8VGN3dQZuZ9Wa1SiIB/F3SDEnjctmQiHg2v34OGJJfDwXmlOzbnMvaK38HSeMkNUpqnDdvXle9BzOzXq/spFRVtGtEzJX0bmCqpMdKV0ZESIquOllETCDNi0JDQ0OXHdfMrLeryZVIRMzNzy8A15LaNJ7P1VTk5xfy5nOBDUt2H5bL2is3M7Nu0u1JRNLqec52JK0O7Ak8DEwGWu6wGgtcl19PBg7Jd2ntDLySq72mAHtKWjs3qO+Zy8zMrJvUojprCHCtpJbz/yki/iZpOnClpMOAp4HP5u1vAD4BNAGvAYcCRMQCSacA0/N2J0fEgu57G2Zmpoje1UTQ0NAQjY2NxXdIya56qvn513Ps4Pg74vjLc/zldTJ+STNKumS8pSfd4mtmZnXGScTMzCrmJGJmZhVzEjEzs4o5iZiZWcWcRMzMrGJOImZmVjEnETMzq5iTiJmZVcxJxMzMKuYkYmZmFXMSMTOzijmJmJlZxZxEzMysYk4iZmZWMScRMzOrmJOImZlVzEnEzMwqVvdJRNJoSY9LapJ0Yq3jMTPrTeo6iUjqA5wL7A1sARwsaYvaRmVm1nvUdRIBdgSaIuLJiHgDmASMqXFMZma9Rt9aB7CChgJzSpabgZ1abyRpHDAuLy6W9HgVY1oPeLHw1lL1Ium8eo4dHH+tOf7aqnb8G7dVWO9JpJCImABM6I5zSWqMiIbuOFdXq+fYwfHXmuOvrVrFX+/VWXOBDUuWh+UyMzPrBvWeRKYDIyWNkLQqcBAwucYxmZn1GnVdnRURSyUdDUwB+gCXRMSsGofVLdVmVVLPsYPjrzXHX1s1iV8RUYvzmpnZSqDeq7PMzKyGnETMzKxiTiJmZlaxum5YtxUnabWIWNJRmVWHpF2BkRHxO0mDgYER8VSt4ypC0lPAOxpVI2KTGoRjNeIksgIkrRkRCyWt09b6iFjQ3TFV4C5guwJlPVYeQ20IJX/PEfFM7SIqRtJ4oAF4L/A7oB/wB+CDtYyrE0o7tvUHPgO0+X+hJ5H0EG0kvxYR8T/dGM4K6Ql/+04iK+ZPwD7ADNIfZek4AgH02F9kkt5DGjZmgKRtWR77msC7ahZYJ0n6BjAeeB54MxcHUA9fBJ8CtgXuA4iIf0tao7YhFRcR81sVnSVpBvDDWsTTCfvk56Py8+/z8xdqEEvFesrfvpPICoiIfSQJ+HA9/PJtZS/gy6Re/meUlC8CvluLgCp0DPDeNr7Q6sEbERGSAkDS6rUOqDMklV6trkK6Munx3ykR8TSApD0iYtuSVSdKug+olyklesTffo//B+/p8pfAX4Gtax1LZ0TERGCipE9HxNW1jmcFzAFeqXUQFbpS0gXAIEmHA18BLqxxTJ1xesnrpcBs4LO1CaUikvTBiLgjL+xCfd1s1CP+9t3ZsAtImgicExHTax1LJSR9EtiSVK8NQEScXLuIipN0MalN4a/AWzcDRMQZ7e7Ug0jaA9iTVJ04JSKm1jikXkPS9sAlwFq56GXgKxFxX+2iKq6n/O37SqRr7AR8UdJs4FXSF0LUQwOdpPNJbSAfBS4CDgTurWlQnfNMfqyaH3UlJ426TBySVgM+DQzn7Q27dfEDJCJmANtIWisv1/xXfSf1iL99X4msAEkbRcQzktocZ7+l7rUnk/RgRPxPyfNA4MaI+FCtY+uMHDcRsbjWsRQlaRHL7xJalXR31qsRsWbtoipO0t9I1SkzgGUt5RFxers79SCShgA/AzaIiL3zrKgfiIiLaxxaXfGVyIr5M7BdRDwt6eqI+HStA6rAf/Lza5I2ABYA69cwnk6RtBXp7pp18vKLwCE9YCDODkXEW3di5Rs0xgA71y6iThsWEaNrHcQKuJR0a/X38vL/AVcAdZFEJN1M2/10du/OOJxEVkzpLb099nbeDlwvaRDwC9IvSkjVWvViAvCtiLgZQNJHSI3Tu9QyqHIk9Y2IpaVlkaoE/pz7jtTL3UF3Sto6Ih6qdSAVWi8irpR0Erw1KviyjnbqQY4red2fVLW4tJ1tq8ZJZMVEO697PEk7AHMi4pS8PBB4CHgMOLOWsXXS6i0JBCAiptXBrbL3AttJOqCkrOUW2ddrE1Jxkh4m9UvoCxwq6UlSw27dtAVmr0pal/x/V9LO9IC7nYrKbTql7pDU7e2ZTiIrZhtJC0n/eQbk17D8P1NPrtu+APg4gKTdgJ8D3wBGkX7dH1i70DrlSUk/YHmHsS8CT9Ywns7Yl+U/Plpukd2vZtEUN5T0d1KXJB0L3Al8B7gO2ETSHcBgUq/7utBqpIyWHyFrtbN59eJww3rvJOmBiNgmvz4XmBcRP8rLMyOiLr4kJK0N/BjYNRfdBvwoIl6qXVTlSWomdfBUq1UBPf/2ZEn3RUTdDIvTmqRfkao730e68p4L3ApcHhEv1jK2zmg1dlnLj5CTI+L27ozDVyK9V5+SuvmPAeNK1tXN30VOFt+sdRyd1AcYyDuTSL14t6RvtbeypyfBiDgOIE+p3UBKKB8BTpL0ckRsUcPwOlRSFT0iL48ltYfMBh7p7njq5svCutzlwC35bqb/kH7BI2kz6qBeWNLkcusjoidXCz1bL30p2lHvSbDFANJYcWvlx79J7YI9Xeuq6FOpYVW0q7N6sdyQuD7w94h4NZdtThqOvEf32pU0jzTsw+XAPbT6QouIW2oRVxGS7m81ZlNdWQmqsyaQRmhYRPrbuRu4uydXgZbqaVXRvhLpxSLi7jbK/q8WsVTgPcAewMHA50lDP1xeD/1DSNWH9azer0A2AlYDniC1hzSThjypFz2qKtpXIlb38vAbBwO/BH4cEefUOKSVmqR16mSunHblzp1bktpDdgG2InW0vSsixtcyto5I+h7wCeBFUkLcLg8EuxkwMSK6dT4aJxGrWzl5fJKUQIYDk4FLImJuLeOy+iFpGGkSsF1I84ysGxGDahtVx3pSVbSTiNUlSZeRfj3eAEyKiIdrHJLVCUnfZPkVyH9JfUZaHg9FxJtldrdWnESsLkl6kzRiMrx9tIB66OhpNSTpDOAO4M6IeLbW8dQ7JxEzM6tYPc3iZWZmPYyTiJmZVcxJxKwVScskzZQ0S9IDkr4tqar/VyT9Mp/vl1U+z/A8Cq9Zl3BnQ7N3+k9Lr19J7wb+RBoeo5r9B8YB60REPc1nYeYrEbNyIuIF0hf80UqGS7pN0n35sQukW44l7d+yn6Q/ShpTeqy8/y8lPSzpIUmfy+WTSWNRzWgpK9nnIUmD8r7zJR1Scr49JPXJx5wu6UFJR5Tse3xJ+Y9bvzdJm0i6Pw/oZ1YRX4mYdSAinpTUB3g38AKwR0S8LmkkaeyuBtKUqv9Lmp1wLVIfhLGtDnUAaZC8bYD1gOmSbo2I/SQtbmfMoztIneGeJs2T8iHgMuADwNeAw4BXImKH3PnyDkl/B0bmx46k254n58H6ngGQ9F5gEvDliHhgxT8l662cRMw6px9wjqRRwDJgc0gDPko6T9Jg0rDcV7eeApc058nlucrqeUm3ADuQetq35zZgN1IS+S0wTtJQ4KWIeFXSnsD/SGoZuXUtUvLYMz/uz+UDc/kzpMmXrgMOiIhuHzrcVi5OImYdkLQJKWG8QGoXeZ50NbEKb5/O9jLSzIoHAYd20elvBY4ijZH0PeBTpKG+b2sJD/hGRExpFfNewKkRcUGr8uGkof6fISU1JxFbIW4TMSsjX1mcD5wTqWfuWqT5QN4EvkSaW6PFpcCxAO38wr8N+FxuxxhMusIoOyd2RMwhVX2NjIgngduB40jJBWAK8DVJ/XK8myvNMT8F+Iqkgbl8aL5JAOANUjI6RNLnC38YZm3wlYjZOw2QNJNUdbWUNH97y2x95wFX5wbuv7F86BUi4nlJjwJ/bue415LaMh4gDdXynYh4rkA897A8Wd1GmoSoZQrUi0iDT96XR6adB+wfEX+X9H7grlTMYtJV0rIc66uS9gGm5vaYspN8mbXHw56YdRFJ7yLNjLddRPT42SHNuoKrs8y6gKSPA48Cv3ECsd7EVyJmZlYxX4mYmVnFnETMzKxiTiJmZlYxJxEzM6uYk4iZmVXMScTMzCr2/wEdC5DNY5uasgAAAABJRU5ErkJggg==\n", + "text/plain": [ + "
      " + ] + }, + "metadata": { + "tags": [], + "needs_background": "light" + } + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "HNPhsx8P2tUH", + "colab_type": "text" + }, + "source": [ + "\n", + "# Spark SQL" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "rHMvBBAh23cw", + "colab_type": "text" + }, + "source": [ + "SQL has been around since the 1970s, and so one can imagine the number of people who made it their bread and butter. As big data came into popularity, the number of professionals with the technical knowledge to deal with it was in shortage. This led to the creation of Spark SQL. To quote the docs:
      \n", + ">Spark SQL is a Spark module for structured data processing. Unlike the basic Spark RDD API, the interfaces provided by Spark SQL provide Spark with more information about the structure of both the data and the computation being performed. Internally, Spark SQL uses this extra information to perform extra optimizations.\n", + "\n", + "Basically, what you need to know is that Spark SQL is used to execute SQL queries on big data. Spark SQL can also be used to read data from Hive tables and views. Let me explain Spark SQL with an example.\n" + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "g2DaK9-D7QkX", + "colab_type": "code", + "outputId": "9899d009-fa16-4fe8-9e94-69fe84c04f5c", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 394 + } + }, + "source": [ + "# Load data\n", + "df = spark.read.csv('reported-crimes.csv',header=True).withColumn('Date',to_timestamp(col('Date'),'MM/dd/yyyy hh:mm:ss a')).filter(col('Date')==lit('2019-07-30'))\n", + "# Register Temporary Table\n", + "df.createOrReplaceTempView(\"temp\")\n", + "# Select all data from temp table\n", + "spark.sql(\"select * from temp\").show()\n", + "# Select count of data in table\n", + "spark.sql(\"select count(*) from temp\").show()" + ], + "execution_count": 0, + "outputs": [ + { + "output_type": "stream", + "text": [ + "+--------+-----------+-------------------+--------------------+----+-------------------+--------------------+--------------------+------+--------+----+--------+----+--------------+--------+------------+------------+----+--------------------+------------+-------------+--------------------+\n", + "| ID|Case Number| Date| Block|IUCR| Primary Type| Description|Location Description|Arrest|Domestic|Beat|District|Ward|Community Area|FBI Code|X Coordinate|Y Coordinate|Year| Updated On| Latitude| Longitude| Location|\n", + "+--------+-----------+-------------------+--------------------+----+-------------------+--------------------+--------------------+------+--------+----+--------+----+--------------+--------+------------+------------+----+--------------------+------------+-------------+--------------------+\n", + "|11895315| JC515309|2019-07-30 00:00:00|097XX S WENTWORTH...|1156| DECEPTIVE PRACTICE|ATTEMPT - FINANCI...| OTHER| false| false|0511| 005| 21| 49| 11| 1176661| 1840424|2019|11/21/2019 03:52:...|41.717467205|-87.628563649|(41.717467205, -8...|\n", + "|11805495| JC406040|2019-07-30 00:00:00|0000X S MICHIGAN AVE|1156| DECEPTIVE PRACTICE|ATTEMPT - FINANCI...| OTHER| false| false|0112| 001| 42| 32| 11| 1177316| 1900297|2019|08/28/2019 04:08:...|41.881749633|-87.624355971|(41.881749633, -8...|\n", + "|11805303| JC405788|2019-07-30 00:00:00|067XX S BLACKSTON...|0890| THEFT| FROM BUILDING| APARTMENT| false| false|0332| 003| 5| 43| 06| 1187366| 1860406|2019|08/28/2019 04:08:...|41.772052629|-87.588722816|(41.772052629, -8...|\n", + "|11798686| JC397805|2019-07-30 00:00:00| 010XX W MADISON ST|1153| DECEPTIVE PRACTICE|FINANCIAL IDENTIT...| null| false| false|1232| 012| 25| 28| 11| 1169700| 1900215|2019|08/21/2019 04:18:...|41.881693864|-87.652323905|(41.881693864, -8...|\n", + "|11787134| JC383936|2019-07-30 00:00:00|110XX S MICHIGAN AVE|1130| DECEPTIVE PRACTICE|FRAUD OR CONFIDEN...| CURRENCY EXCHANGE| false| false|0513| 005| 9| 49| 11| 1178795| 1831638|2019|08/11/2019 04:00:...|41.693309015|-87.621013915|(41.693309015, -8...|\n", + "|11781214| JC376972|2019-07-30 00:00:00| 015XX N LEAVITT ST|0820| THEFT| $500 AND UNDER| STREET| false| false|1424| 014| 1| 24| 06| 1161451| 1910391|2019|08/07/2019 03:58:...|41.909793259|-87.682330427|(41.909793259, -8...|\n", + "|11775928| JC370624|2019-07-30 00:00:00|079XX S EBERHART AVE|0820| THEFT| $500 AND UNDER|VEHICLE NON-COMME...| false| false|0624| 006| 6| 44| 06| 1180949| 1852510|2019|08/06/2019 04:17:...|41.750535224|-87.612487858|(41.750535224, -8...|\n", + "|11776857| JC371434|2019-07-30 00:00:00|070XX S EBERHART AVE|5002| OTHER OFFENSE|OTHER VEHICLE OFF...| STREET| false| false|0322| 003| 6| 69| 26| 1180795| 1858429|2019|08/06/2019 04:17:...|41.766781131|-87.612870529|(41.766781131, -8...|\n", + "|11776516| JC371429|2019-07-30 00:00:00| 057XX N CLARK ST|0281|CRIM SEXUAL ASSAULT| NON-AGGRAVATED| ALLEY| false| false|2013| 020| 48| 77| 02| 1164693| 1938570|2019|08/06/2019 04:17:...|41.987049672|-87.669619077|(41.987049672, -8...|\n", + "|11776159| JC370564|2019-07-30 00:00:00| 066XX N ROCKWELL ST|0620| BURGLARY| UNLAWFUL ENTRY| APARTMENT| false| false|2412| 024| 50| 2| 05| 1157787| 1943931|2019|08/06/2019 04:17:...|42.001904519|-87.694872497|(42.001904519, -8...|\n", + "+--------+-----------+-------------------+--------------------+----+-------------------+--------------------+--------------------+------+--------+----+--------+----+--------------+--------+------------+------------+----+--------------------+------------+-------------+--------------------+\n", + "\n", + "+--------+\n", + "|count(1)|\n", + "+--------+\n", + "| 10|\n", + "+--------+\n", + "\n" + ], + "name": "stdout" + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "6i32WE8j_ec8", + "colab_type": "text" + }, + "source": [ + "As you can see, we registered the dataframe as temporary table and then ran basic SQL queries on it. How amazing is that?!
      \n", + "If you are a person who is more comfortable with SQL, then this feature is truly a blessing for you! But this raises a question: \n", + "> *Should I just keep using Spark SQL all the time?*\n", + "\n", + "And the answer is, _**it depends**_.
      \n", + "So basically, the different functions acts in differnet ways, and depending upon the type of action you are trying to do, the speed at which it completes execution also differs. But as time progress, this feature is getting better and better, so hopefully the difference should be a small margin. There are plenty of analysis done on this, but nothing has a definite answer yet. You can read this [comparative study done by horton works](https://community.cloudera.com/t5/Community-Articles/Spark-RDDs-vs-DataFrames-vs-SparkSQL/ta-p/246547) or the answer to this [stackoverflow question](https://stackoverflow.com/questions/45430816/writing-sql-vs-using-dataframe-apis-in-spark-sql) if you are still curious about it." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "x62BiCgBMOtq", + "colab_type": "text" + }, + "source": [ + "\n", + "# RDD" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "VGXK6uEuUKRh", + "colab_type": "text" + }, + "source": [ + "> With map, you define a function and then apply it record by record. Flatmap returns a new RDD by first applying a function to all of the elements in RDDs and then flattening the result. Filter, returns a new RDD. Meaning only the elements that satisfy a condition. With reduce, we are taking neighboring elements and producing a single combined result.\n", + "For example, let's say you have a set of numbers. You can reduce this to its sum by providing a function that takes as input two values and reduces them to one. \n", + "\n", + "Some of the reasons you would use a dataframe over RDD are:\n", + "1. It's ability to represnt data as rows and columns. But this also means it can only hold structred and semi-structured data.\n", + "2. It allows processing data in different formats (AVRO, CSV, JSON, and storage system HDFS, HIVE tables, MySQL).\n", + "3. It's superior job Optimization capability.\n", + "4. DataFrame API is very easy to use.\n", + "\n", + "\n", + "\n" + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "0_WvAgyvR7m6", + "colab_type": "code", + "outputId": "73daad9c-5f3a-4529-dd17-11f36eff6689", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 71 + } + }, + "source": [ + "psrdd = sc.textFile('police-station.csv')\n", + "print(psrdd.first())\n", + "ps_header = psrdd.first()\n", + "ps_rest = psrdd.filter(lambda line: line!=ps_header)\n", + "print(ps_rest.first())" + ], + "execution_count": 0, + "outputs": [ + { + "output_type": "stream", + "text": [ + "DISTRICT,DISTRICT NAME,ADDRESS,CITY,STATE,ZIP,WEBSITE,PHONE,FAX,TTY,X COORDINATE,Y COORDINATE,LATITUDE,LONGITUDE,LOCATION\n", + "1,Central,1718 S State St,Chicago,IL,60616,http://home.chicagopolice.org/community/districts/1st-district-central/,312-745-4290,312-745-3694,312-745-3693,1176569.052,1891771.704,41.85837259,-87.62735617,\"(41.8583725929, -87.627356171)\"\n" + ], + "name": "stdout" + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "P65eAFO3Mkdd", + "colab_type": "text" + }, + "source": [ + "**How many police stations are there?**" + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "Vi03EU0CMSmO", + "colab_type": "code", + "outputId": "3f5f62da-b73d-44e7-f6fc-0b7e493635e2", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 34 + } + }, + "source": [ + "ps_rest.map(lambda line: line.split(\",\")).count()" + ], + "execution_count": 0, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "24" + ] + }, + "metadata": { + "tags": [] + }, + "execution_count": 65 + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "3c4bci70MnlQ", + "colab_type": "text" + }, + "source": [ + "**Display the District ID, District name, Address and Zip for the police station with District ID 7**" + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "fWFpo_WxMnvm", + "colab_type": "code", + "outputId": "9ff7afd8-c0aa-4e84-ba7d-ac07962e3c1b", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 34 + } + }, + "source": [ + "# District is column 0\n", + "(ps_rest.filter(lambda line: line.split(\",\")[0]=='7').\n", + " map(lambda line: (line.split(\",\")[0],\n", + " line.split(\",\")[1],\n", + " line.split(\",\")[2],\n", + " line.split(\",\")[5])).collect())" + ], + "execution_count": 0, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "[('7', 'Englewood', '1438 W 63rd St', '60636')]" + ] + }, + "metadata": { + "tags": [] + }, + "execution_count": 66 + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "ZYmb5FscMph3", + "colab_type": "text" + }, + "source": [ + "**Police stations 10 and 11 are geographically close to each other. Display the District ID, District name, address and zip code**" + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "6ZcRIX3mMquF", + "colab_type": "code", + "outputId": "c0575516-f533-4c74-b14e-73b903577fd3", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 51 + } + }, + "source": [ + "# District is column 0\n", + "(ps_rest.filter(lambda line: line.split(\",\")[0] in ['10', '11']).\n", + " map(lambda line: (line.split(\",\")[0],\n", + " line.split(\",\")[1],\n", + " line.split(\",\")[2],\n", + " line.split(\",\")[5])).collect())" + ], + "execution_count": 0, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "[('11', 'Harrison', '3151 W Harrison St', '60612'),\n", + " ('10', 'Ogden', '3315 W Ogden Ave', '60623')]" + ] + }, + "metadata": { + "tags": [] + }, + "execution_count": 67 + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "3wn2zXe7TbI3", + "colab_type": "text" + }, + "source": [ + "\n", + "# User-Defined Functions (UDF)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "w0YWspcTRrin", + "colab_type": "text" + }, + "source": [ + "PySpark User-Defined Functions (UDFs) help you convert your python code into a scalable version of itself. It comes in handy more than you can imagine, but beware, as the performance is less when you compare it with pyspark functions. You can view examples of how UDF works [here](https://docs.databricks.com/spark/latest/spark-sql/udf-python.html). What I will give in this section is some theory on how it works, and why it is slower.\n", + "\n", + "When you try to run a UDF in PySpark, each executor creates a python process. Data will be serialised and deserialised between each executor and python. This leads to lots of performance impact and overhead on spark jobs, making it less efficent than using spark dataframes. Apart from this, sometimes you might have memory issues while using UDFs. The Python worker consumes huge off-heap memory and so it often leads to memoryOverhead, thereby failing your job. Keeping these in mind, I wouldn't recommend using them, but at the end of the day, your choice." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "yv7ODDTQRwVt", + "colab_type": "text" + }, + "source": [ + "\n", + "# Common Questions" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "6z9gkoE2R1m1", + "colab_type": "text" + }, + "source": [ + "\n", + "## Recommended IDE" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "BpMTbRggR5Z8", + "colab_type": "text" + }, + "source": [ + "I personally prefer [PyCharm](https://www.jetbrains.com/pycharm/) while coding in Python/PySpark. It's based on IntelliJ IDEA so it has a lot of features! And the main advantage I have felt is the ease of installing PySpark and other packages. You can customize it with themes and plugins, and it lets you enhance productivity while coding by providing some features like suggestions, local VCS etc." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "VZ1bYvF8R8Dc", + "colab_type": "text" + }, + "source": [ + "\n", + "## Submitting a Spark Job" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "3EQWnq23SCbE", + "colab_type": "text" + }, + "source": [ + "The python syntax for running jobs is: `python .py ...`\n", + "
      But when you submit a spark job you have to use spark-submit to run the application.\n", + "\n", + "Here is a simple example of a spark-submit command:\n", + "`spark-submit filename.py --named_argument 'arguemnt value'`
      \n", + "Here, named_argument is an argument that you are reading from inside your script.\n", + "\n", + "There are other options you can pass in the command, like:
      \n", + "`--py-files` which helps you pass a python file to read in your file,
      \n", + "`--files` which helps pass other files like txt or config,
      \n", + "`--deploy-mode` which tells wether to deploy your worker node on cluster or locally
      \n", + "`--conf` which helps pass different configurations, like memoryOverhead, dynamicAllocation etc.\n", + "\n", + "There is an [entire page](https://spark.apache.org/docs/latest/submitting-applications.html) in spark documentation dedicated to this. I highly recommend you go through it once." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "oVwGYAZZiyGV", + "colab_type": "text" + }, + "source": [ + "\n", + "## Creating Dataframes" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "TvndhPjoi0er", + "colab_type": "text" + }, + "source": [ + "When getting started with dataframes, the most common question is: *'How do I create a dataframe?'*
      \n", + "Below, you can see how to create three kinds of dataframes:" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "QXmRD3hHlM-f", + "colab_type": "text" + }, + "source": [ + "### Create a totally empty dataframe" + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "ktkb6s-kjtgG", + "colab_type": "code", + "outputId": "0a81f022-0d22-4fe9-d34e-1663467591f3", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 102 + } + }, + "source": [ + "from pyspark.sql.types import StructType\n", + "#Create empty df\n", + "schema = StructType([])\n", + "empty = spark.createDataFrame(sc.emptyRDD(), schema)\n", + "empty.show()" + ], + "execution_count": 0, + "outputs": [ + { + "output_type": "stream", + "text": [ + "++\n", + "||\n", + "++\n", + "++\n", + "\n" + ], + "name": "stdout" + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "mg5K3nz_lSDe", + "colab_type": "text" + }, + "source": [ + "### Create an empty dataframe with header" + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "9raf4CkRjuTr", + "colab_type": "code", + "outputId": "6109efd0-c4ca-49c4-b724-1f6066eed7e2", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 102 + } + }, + "source": [ + " from pyspark.sql.types import StructType, StructField\n", + "#Create empty df with header\n", + "schema_header = StructType([StructField(\"name\", StringType(), True)])\n", + "empty_with_header = spark.createDataFrame(sc.emptyRDD(), schema_header)\n", + "empty_with_header.show()" + ], + "execution_count": 0, + "outputs": [ + { + "output_type": "stream", + "text": [ + "+----+\n", + "|name|\n", + "+----+\n", + "+----+\n", + "\n" + ], + "name": "stdout" + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "Y1ZNOx7ilUnd", + "colab_type": "text" + }, + "source": [ + "### Create a dataframe with header and data" + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "TvzyL46QkJBl", + "colab_type": "code", + "outputId": "952991b0-fca9-4533-ed29-0fb42cdcd8dd", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 153 + } + }, + "source": [ + "from pyspark.sql import Row\n", + "mylist = [\n", + " {\"name\":'Alice',\"age\":13},\n", + " {\"name\":'Jacob',\"age\":24},\n", + " {\"name\":'Betty',\"age\":135},\n", + "]\n", + "spark.createDataFrame(Row(**x) for x in mylist).show()" + ], + "execution_count": 0, + "outputs": [ + { + "output_type": "stream", + "text": [ + "+---+-----+\n", + "|age| name|\n", + "+---+-----+\n", + "| 13|Alice|\n", + "| 24|Jacob|\n", + "|135|Betty|\n", + "+---+-----+\n", + "\n" + ], + "name": "stdout" + } + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "VnRMckA5nLoJ", + "colab_type": "code", + "outputId": "35cdaa1b-e984-496d-b4e7-80b5cfdc766b", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 153 + } + }, + "source": [ + "# You can achieve the same using this - note that we are using spark context here, not a spark session\n", + "from pyspark.sql import Row\n", + "df = sc.parallelize([\n", + " Row(name='Alice', age=13),\n", + " Row(name='Jacob', age=24),\n", + " Row(name='Betty', age=135)]).toDF()\n", + "df.show()" + ], + "execution_count": 0, + "outputs": [ + { + "output_type": "stream", + "text": [ + "+---+-----+\n", + "|age| name|\n", + "+---+-----+\n", + "| 13|Alice|\n", + "| 24|Jacob|\n", + "|135|Betty|\n", + "+---+-----+\n", + "\n" + ], + "name": "stdout" + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "f3crkAQVlxKp", + "colab_type": "text" + }, + "source": [ + "\n", + "## Drop Duplicates" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "4IHrYEwHmBcc", + "colab_type": "text" + }, + "source": [ + "As mentioned earlier, there are two wasy to remove duplicates from a dataframe. We have already seen the usage of distinct under Get Distinct Rows section. \n", + "I will expalin how to use the `dropDuplicates()` function to achieve the same. \n", + "\n", + "> `drop_duplicates()` is an alias for `dropDuplicates()`" + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "wOuHRAPJmWen", + "colab_type": "code", + "outputId": "a2b86418-8454-4c74-99bc-0836a8ae4b3e", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 136 + } + }, + "source": [ + "from pyspark.sql import Row\n", + "from pyspark.sql import Row\n", + "mylist = [\n", + " {\"name\":'Alice',\"age\":5,\"height\":80},\n", + " {\"name\":'Jacob',\"age\":24,\"height\":80},\n", + " {\"name\":'Alice',\"age\":5,\"height\":80}\n", + "]\n", + "df = spark.createDataFrame(Row(**x) for x in mylist)\n", + "df.dropDuplicates().show()" + ], + "execution_count": 0, + "outputs": [ + { + "output_type": "stream", + "text": [ + "+---+------+-----+\n", + "|age|height| name|\n", + "+---+------+-----+\n", + "| 5| 80|Alice|\n", + "| 24| 80|Jacob|\n", + "+---+------+-----+\n", + "\n" + ], + "name": "stdout" + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "zMv7A-2Hnmjh", + "colab_type": "text" + }, + "source": [ + "`dropDuplicates()` can also take in an optional parameter called *subset* which helps specify the columns on which the duplicate check needs to be done on." + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "SHnFylV1n8to", + "colab_type": "code", + "outputId": "a2a37b6b-fd35-47ed-a55a-6fb48032ab87", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 119 + } + }, + "source": [ + "df.dropDuplicates(subset=['height']).show()" + ], + "execution_count": 0, + "outputs": [ + { + "output_type": "stream", + "text": [ + "+---+------+-----+\n", + "|age|height| name|\n", + "+---+------+-----+\n", + "| 5| 80|Alice|\n", + "+---+------+-----+\n", + "\n" + ], + "name": "stdout" + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "bAS4DKxjqI7H", + "colab_type": "text" + }, + "source": [ + "\n", + "## Fine Tuning a Spark Job" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "2d9pLlDl76dM", + "colab_type": "text" + }, + "source": [ + "Before we begin, please note that this entire section is written purely based on experience. It might differ with use cases, but it will help you get a better understanding of what you should be looking for, or act as a guidance to achieve your aim.\n", + "\n", + ">Spark Performance Tuning refers to the process of adjusting settings to record for memory, cores, and instances used by the system. This process guarantees that the Spark has a flawless performance and also prevents bottlenecking of resources in Spark.\n", + "\n", + "Considering you are using Amazon EMR to execute your spark jobs, there are two aspects you need to take care of:\n", + "1. EMR Sizing\n", + "2. Spark Configurations\n", + "\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "vIbXZT29JxmG", + "colab_type": "text" + }, + "source": [ + "\n", + "### EMR Sizing" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "2Rv2SM_-KA8W", + "colab_type": "text" + }, + "source": [ + "Sizing your EMR is extremely important, as this affects the efficency of your spark jobs. Apart from the cost factor, the maximum number of nodes and memory your job can use will be decided by this. If you spin up a EMR with high specifications, that obviously means you are paying more for it, so we should ideally utilize it to the max. These are the guidelines that I follow to make sure the EMR is rightly sized:\n", + "\n", + "1. Size of the input data (include all the input data) on the disk.\n", + "2. Whether the jobs have transformations or just a straight pass through.
      Assess the joins and the complex joins involved.\n", + "3. Size of the output data on the disk.\n", + "\n", + "Look at the above criteria against the memory you need to process, and the disk space you would need. Start with a small configuration, and keep adding nodes to arrive at an optimal configuration. In case you are wondering about the *Execution time vs EMR configuration* factor, please understand that it is okay for a job to run longer, rather than adding more resources to the cluster. For example, it is okay to run a job for 40 mins job on a 5 node cluster, rather than running a job in 10 mins on a 15 node cluster.\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "pmeFEgv6QTST", + "colab_type": "text" + }, + "source": [ + "Another thing you need to know about EMRs, are the different kinds of EC2 instance types provided by Amazon. I will briefly talk about them, but I strongly recommend you to read more about it from the [official documentation](https://aws.amazon.com/ec2/instance-types/). There are 5 types of instance classes. Based on the job you want to run, you can decide which one to use:\n", + "\n", + ">Instance Class | Description\n", + ">--- | ---\n", + ">General purpose | Balance of compute, memory and networking resources\n", + ">Compute optimized | Ideal for compute bound applications that benefit from high performance processors\n", + ">Memory optimized | Designed to deliver fast performance for workloads that process large data sets in memory\n", + ">Storage optimized | For workloads that require high, sequential read and write access to very large data sets on local storage\n", + ">GPU instances | Use hardware accelerators, or co-processors, to perform high demanding functions, more efficiently than is possible in software running on CPUs\n", + "\n", + "The configuration (memory, storage, cpu, network performance) will differ based on the instance class you choose.
      \n", + "To help make life easier, here is what I do when I get into a predicament about which one to go with:
      \n", + " 1. Visit [ec2instances](www.https://www.ec2instances.info/)\n", + " 2. Choose the EC2 instances in question \n", + " 3. Click on compare selected\n", + "\n", + "This will easily help you undesrstand what you are getting into, and thereby help you make the best choice! The site was built by [Garret Heaton](https://github.com/powdahound)(founder of Swoot), and has helped me countless number of times to make an informed decision." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "snACMwZug5Yn", + "colab_type": "text" + }, + "source": [ + "\n", + "### Spark Configurations" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "uFJNpK06hnpo", + "colab_type": "text" + }, + "source": [ + "There are a ton of [configurations](https://spark.apache.org/docs/latest/configuration.html) that you can tweak when it comes to Spark. Here, I will be noting down some of the configurations which I use, which have worked well for me. Alright! let's get into it!" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "dABRu9eokZxw", + "colab_type": "text" + }, + "source": [ + "#### Job Scheduling" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "8fRFs6atkdxS", + "colab_type": "text" + }, + "source": [ + "When you submit your job in a cluster, it will be given to Spark Schedulers, which is responsible for materializing a logical plan for your job. There are two types of [job scheduling](https://spark.apache.org/docs/latest/job-scheduling.html): \n", + "1. FIFO
      \n", + "By default, Spark’s scheduler runs jobs in FIFO fashion. Each job is divided into stages (e.g. map and reduce phases), and the first job gets priority on all available resources while its stages have tasks to launch, then the second job gets priority, etc. If the jobs at the head of the queue don’t need to use the whole cluster, later jobs can start to run right away, but if the jobs at the head of the queue are large, then later jobs may be delayed significantly. \n", + "2. FAIR
      \n", + "The fair scheduler supports grouping jobs into pools and setting different scheduling options (e.g. weight) for each pool. This can be useful to create a high-priority pool for more important jobs, for example, or to group the jobs of each user together and give users equal shares regardless of how many concurrent jobs they have instead of giving jobs equal shares. This approach is modeled after the Hadoop Fair Scheduler.\n", + "\n", + "> I personally prefer using the FAIR mode, and this can be set by adding `.config(\"spark.scheduler.mode\", \"FAIR\")` when you create your SparkSession.\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "xrgbAiZHnq_U", + "colab_type": "text" + }, + "source": [ + "#### Serializer" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "N7ZYGGcYsRzB", + "colab_type": "text" + }, + "source": [ + "We have two types of [serializers](https://spark.apache.org/docs/latest/tuning.html#data-serialization) available: \n", + "1. Java serialization \n", + "2. Kryo serialization\n", + "\n", + "Kryo is significantly faster and more compact than Java serialization (often as much as 10x), but does not support all Serializable types and requires you to register the classes you’ll use in the program in advance for best performance.\n", + "\n", + "Java serialization is used by default because if you have custom class that extends Serializable it can be easily used. You can also control the performance of your serialization more closely by extending java.io.Externalizable\n", + "\n", + "> The general recommendation is to use Kyro as the serializer whenver possible, as it leads to much smaller sizes than Java serialization. It can be added by using `.config(\"spark.serializer\", \"org.apache.spark.serializer.KryoSerializer\")` when you create your SparkSession.\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "244El832wT8f", + "colab_type": "text" + }, + "source": [ + "#### Shuffle Behaviour" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "XWsBFnA8xpeF", + "colab_type": "text" + }, + "source": [ + "It is generally a good idea to compress the output file after the map phase. The `spark.shuffle.compress` property decides whether to do the compression or not. The compression used is `spark.io.compression.codec`.\n", + "\n", + "> The property can be added by using `.config(\"spark.shuffle.compress\", \"true\")` when you create your SparkSession." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "H5cFCvbczHz_", + "colab_type": "text" + }, + "source": [ + "#### Compression and Serialization" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "LHU5lfyFzKw9", + "colab_type": "text" + }, + "source": [ + "There are 4 defaiult codecs spark provides to compress internal data such as RDD partitions, event log, broadcast variables and shuffle outputs. They are: \n", + "\n", + "1. lz4\n", + "2. lzf\n", + "3. snappy\n", + "4. zstd\n", + "\n", + "> The decision on which to use rests upon the use case. I generally use the `snappy` compression. Google created Snappy because they needed something that offered very fast compression at the expense of final size. Snappy is fast, stable and free, but it increases the size more than the other codecs. At the same time, since compute costs will be less, it seems like balanced trade off. The property can be added by using `.config(\"spark.io.compression.codec\", \"snappy\")` when you create your SparkSession.\n", + "\n", + "This [session](https://databricks.com/session/best-practice-of-compression-decompression-codes-in-apache-spark) explains the best practice of compression/decompression codes in Apache Spark. I recommend you to take a look at it before taking a decision." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "rdGARl-D3n-l", + "colab_type": "text" + }, + "source": [ + "#### Scheduling" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "Cm0ExAoS4RU6", + "colab_type": "text" + }, + "source": [ + "The property `spark.speculation` performs speculative execution of tasks. This means if one or more tasks are running slowly in a stage, they will be re-launched. Speculative execution will not stop the slow running task but it launches the new task in parallel.\n", + "\n", + "> I usually disable this option by adding `.config(\"spark.speculation\", \"false\") ` when I create the SparkSession. " + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "aaxfGqYZ6Iqz", + "colab_type": "text" + }, + "source": [ + "#### Application Properties" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "ERovEKOU6TNE", + "colab_type": "text" + }, + "source": [ + "There are mainly two application properties that you should know about:\n", + "\n", + "1. spark.driver.memoryOverhead - The amount of off-heap memory to be allocated per driver in cluster mode, in MiB unless otherwise specified. This is memory that accounts for things like VM overheads, interned strings, other native overheads, etc. This tends to grow with the container size (typically 6-10%). This option is currently supported on YARN and Kubernetes.\n", + "\n", + "2. spark.executor.memoryOverhead - The amount of off-heap memory to be allocated per executor, in MiB unless otherwise specified. This is memory that accounts for things like VM overheads, interned strings, other native overheads, etc. This tends to grow with the executor size (typically 6-10%). This option is currently supported on YARN and Kubernetes.\n", + "\n", + "> If you ever face an issue like `Container killed by YARN for exceeding memory limits`, know that it is because you have not specified enough memory Overhead for your job to successfully execute. The default value for Overhead is 10% of avaialbe memory (driver/executor sepearte), with minimum of 384.\n", + "\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "BG09dDdL6Tvt", + "colab_type": "text" + }, + "source": [ + "#### Dynamic Allocation" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "m_lb-JI78CVT", + "colab_type": "text" + }, + "source": [ + "Lastly, I want to talk about Dynamic Allocation. This is a feature I constantly use while executing my jobs. This property is by defualt set to False. As the name suggests, it sets whether to use dynamic resource allocation, which scales the number of executors registered with this application up and down based on the workload. Truly a wonderful feature, and the greatest benefit of using it is that it will help make the best use of all the resources you have! The disadvantage of this feature is that it does not shine well when you have to execute tasks in parallel. Since most of the resources will be used by the first task, the second one will have to wait till some resource gets released. At the same time, if both get submitted at the exact same time, the resources will be shared between them, although not equally. Also, it is not guaranteed to *always* use the most optimal configurations. But in all my tests, the results have been great! \n", + "\n", + "> If you are planning on using this feature, you can pass the configurations as required through the spark-submit command. The four configurations which you will have to keep in mind are:
      \n", + "```\n", + "--conf spark.dynamicAllocation.enabled=true\n", + "--conf spark.dynamicAllocation.initialExecutors\n", + "--conf spark.dynamicAllocation.minExecutors\n", + "--conf spark.dynamicAllocation.maxExecutors\n", + "```\n", + "\n", + "You can read more about this feature [here](https://https://spark.apache.org/docs/latest/configuration.html#dynamic-allocation) and [here](https://stackoverflow.com/questions/40200389/how-to-execute-spark-programs-with-dynamic-resource-allocation).\n", + "\n", + "\n", + "\n" + ] + } + ] +} \ No newline at end of file