swift_data.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Retrieving the Dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Introduction\n",
    "\n",
    "This section of the project focuses on retrieving Taylor Swift's music data using the **Spotipy** package. To achieve this, I followed the guide *[How to Extract Any Artist’s Data Using Spotify’s API, Python, and Spotipy](https://betterprogramming.pub/how-to-extract-any-artists-data-using-spotify-s-api-python-and-spotipy-4c079401bc37)* and referred to the Spotipy [documentation](https://spotipy.readthedocs.io/en/2.18.0/).\n",
    "\n",
    "Once the data was retrieved, I performed some data cleaning to organize the dataset and prepare it for analysis.\n",
    "\n",
    "**NOTE:** The data retrieval process requires playlists as inputs with a maximum of 100 songs. Thus, I divided Taylor Swift's discography into three different playlists. \n",
    "\n",
    "**UPDATED:**\n",
    "\n",
    "- 02/05/2023 - Added *Midnights (3am Edition)* album"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.1 Importing Packages\n",
    "\n",
    "These are the packages used in this section of the project."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import required packages\n",
    "from spotipy.oauth2 import SpotifyClientCredentials\n",
    "import spotipy\n",
    "import pandas as pd\n",
    "import os"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.2 Defining Variables\n",
    "\n",
    "In order to retrieve data, you must have a **Spotify Developer** [account](https://developer.spotify.com).\n",
    "\n",
    "Using my developer account, I retrieved the **Client ID** and the **Client Secret ID** and stored them in my local environment to keep them hidden. I also stored my Spotify username and playlist IDs as variables. Playlist IDs can be retrieved from the link to the respective playlist.\n",
    "\n",
    "To set environemntal variables, you must run the following code in the terminal, replacing **'key'** and **'secret_key'** with the Client ID and Client Secret ID, respectively:\n",
    "\n",
    "`export SPOTIFY_CLIENT_KEY=key`\n",
    "\n",
    "`export SPOTIFY_CLIENT_SECRET=secret_key`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Obtain Spotify client IDs, username, and playlist IDs\n",
    "CLIENT_ID = os.environ.get(\"SPOTIFY_CLIENT_KEY\")\n",
    "CLIENT_SECRET = os.environ.get(\"SPOTIFY_CLIENT_SECRET\")\n",
    "SPOTIFY_USERNAME = \"johncarlomaula\"\n",
    "PLAYLIST_IDS = [\"4ea0fs3hTNhM4DEyTxgbLz\", \"0T7Q1ITkVJUESKSj1lw2l1\", \"2JqK8hHOs1wQ2tgtLCZ9KA\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Defining Functions to Retrieve Data\n",
    "\n",
    "I defined the functions that will take the playlist IDs as inputs and retrieve the musical features of the songs contained in those playlists."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.1 Retrieving Track IDs\n",
    "\n",
    "This function iterates through the songs in a playlist and retrieves their respective IDs. It returns a list of song IDs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define function to retrieve track IDs from playlist\n",
    "def get_track_IDs(user, playlist_id):\n",
    "    ids = []\n",
    "    playlist = sp.user_playlist(user, playlist_id)\n",
    "    for item in playlist['tracks']['items']:\n",
    "        track = item['track']\n",
    "        ids.append(track['id'])\n",
    "    return ids"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.2 Extracting Track Features\n",
    "\n",
    "This function uses the track ID and retrieves the Spotify features of that track. It returns a tuple of the track features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define function to extract track metadata and track features\n",
    "def get_track_features(id):\n",
    "    meta = sp.track(id)\n",
    "    features = sp.audio_features(id)\n",
    "\n",
    "    # Metadata\n",
    "    name = meta['name']\n",
    "    album = meta['album']['name']\n",
    "    artist = meta['album']['artists'][0]['name']\n",
    "    release_date = meta['album']['release_date']\n",
    "    length = meta['duration_ms']\n",
    "    popularity = meta['popularity']\n",
    "\n",
    "    # Features\n",
    "    acousticness = features[0]['acousticness']\n",
    "    danceability = features[0]['danceability']\n",
    "    energy = features[0]['energy']\n",
    "    instrumentalness = features[0]['instrumentalness']\n",
    "    liveness = features[0]['liveness']\n",
    "    loudness = features[0]['loudness']\n",
    "    speechiness = features[0]['speechiness']\n",
    "    tempo = features[0]['tempo']\n",
    "    time_signature = features[0]['time_signature']\n",
    "    valence = features[0]['valence']\n",
    "\n",
    "    track = [name, album, artist, release_date, length, popularity, danceability, acousticness, energy,\n",
    "             instrumentalness, liveness, loudness, speechiness, tempo, time_signature, valence]\n",
    "    return track"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Retrieving the Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.1 Spotify Authentication\n",
    "\n",
    "Using my Client IDs, I authenticated my account to gain access to the Spotify data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Call spotify client credentials\n",
    "client_credentials_manager = SpotifyClientCredentials(CLIENT_ID, CLIENT_SECRET)\n",
    "sp = spotipy.Spotify(client_credentials_manager=client_credentials_manager)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.2 Retrieving the Data\n",
    "\n",
    "I called the functions defined in the previous section in order to retrieve the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Call function to retrieve Spotify IDs\n",
    "IDs = []\n",
    "for id in PLAYLIST_IDS:\n",
    "    IDs += get_track_IDs(SPOTIFY_USERNAME, id)\n",
    "    \n",
    "# Extract features of each song\n",
    "tracks = []\n",
    "for id in IDs:\n",
    "    track = get_track_features(id)\n",
    "    tracks.append(track)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.3 Creating the Dataset\n",
    "\n",
    "Finally, I stored the dataset inside a dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create dataframe\n",
    "df = pd.DataFrame(tracks, columns = ['name', 'album', 'artist', 'release_date', 'length', 'popularity',\n",
    "                                    'danceability', 'acousticness', 'energy', 'instrumentalness', 'liveness',\n",
    "                                    'loudness', 'speechiness', 'tempo', 'time_signature', 'valence'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Finalizing the Dataset\n",
    "\n",
    "Before exporting the dataset, I wanted to perform some simple data cleaning to organize the dataset for analysis."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.1 Examining the Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>album</th>\n",
       "      <th>artist</th>\n",
       "      <th>release_date</th>\n",
       "      <th>length</th>\n",
       "      <th>popularity</th>\n",
       "      <th>danceability</th>\n",
       "      <th>acousticness</th>\n",
       "      <th>energy</th>\n",
       "      <th>instrumentalness</th>\n",
       "      <th>liveness</th>\n",
       "      <th>loudness</th>\n",
       "      <th>speechiness</th>\n",
       "      <th>tempo</th>\n",
       "      <th>time_signature</th>\n",
       "      <th>valence</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Tim McGraw</td>\n",
       "      <td>Taylor Swift</td>\n",
       "      <td>Taylor Swift</td>\n",
       "      <td>2006-10-24</td>\n",
       "      <td>232106</td>\n",
       "      <td>60</td>\n",
       "      <td>0.580</td>\n",
       "      <td>0.575</td>\n",
       "      <td>0.491</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.1210</td>\n",
       "      <td>-6.462</td>\n",
       "      <td>0.0251</td>\n",
       "      <td>76.009</td>\n",
       "      <td>4</td>\n",
       "      <td>0.425</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Picture To Burn</td>\n",
       "      <td>Taylor Swift</td>\n",
       "      <td>Taylor Swift</td>\n",
       "      <td>2006-10-24</td>\n",
       "      <td>173066</td>\n",
       "      <td>64</td>\n",
       "      <td>0.658</td>\n",
       "      <td>0.173</td>\n",
       "      <td>0.877</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0962</td>\n",
       "      <td>-2.098</td>\n",
       "      <td>0.0323</td>\n",
       "      <td>105.586</td>\n",
       "      <td>4</td>\n",
       "      <td>0.821</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Teardrops On My Guitar - Radio Single Remix</td>\n",
       "      <td>Taylor Swift</td>\n",
       "      <td>Taylor Swift</td>\n",
       "      <td>2006-10-24</td>\n",
       "      <td>203040</td>\n",
       "      <td>62</td>\n",
       "      <td>0.621</td>\n",
       "      <td>0.288</td>\n",
       "      <td>0.417</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.1190</td>\n",
       "      <td>-6.941</td>\n",
       "      <td>0.0231</td>\n",
       "      <td>99.953</td>\n",
       "      <td>4</td>\n",
       "      <td>0.289</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>A Place in this World</td>\n",
       "      <td>Taylor Swift</td>\n",
       "      <td>Taylor Swift</td>\n",
       "      <td>2006-10-24</td>\n",
       "      <td>199200</td>\n",
       "      <td>51</td>\n",
       "      <td>0.576</td>\n",
       "      <td>0.051</td>\n",
       "      <td>0.777</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.3200</td>\n",
       "      <td>-2.881</td>\n",
       "      <td>0.0324</td>\n",
       "      <td>115.028</td>\n",
       "      <td>4</td>\n",
       "      <td>0.428</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Cold As You</td>\n",
       "      <td>Taylor Swift</td>\n",
       "      <td>Taylor Swift</td>\n",
       "      <td>2006-10-24</td>\n",
       "      <td>239013</td>\n",
       "      <td>52</td>\n",
       "      <td>0.418</td>\n",
       "      <td>0.217</td>\n",
       "      <td>0.482</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.1230</td>\n",
       "      <td>-5.769</td>\n",
       "      <td>0.0266</td>\n",
       "      <td>175.558</td>\n",
       "      <td>4</td>\n",
       "      <td>0.261</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                          name         album        artist  \\\n",
       "0                                   Tim McGraw  Taylor Swift  Taylor Swift   \n",
       "1                              Picture To Burn  Taylor Swift  Taylor Swift   \n",
       "2  Teardrops On My Guitar - Radio Single Remix  Taylor Swift  Taylor Swift   \n",
       "3                        A Place in this World  Taylor Swift  Taylor Swift   \n",
       "4                                  Cold As You  Taylor Swift  Taylor Swift   \n",
       "\n",
       "  release_date  length  popularity  danceability  acousticness  energy  \\\n",
       "0   2006-10-24  232106          60         0.580         0.575   0.491   \n",
       "1   2006-10-24  173066          64         0.658         0.173   0.877   \n",
       "2   2006-10-24  203040          62         0.621         0.288   0.417   \n",
       "3   2006-10-24  199200          51         0.576         0.051   0.777   \n",
       "4   2006-10-24  239013          52         0.418         0.217   0.482   \n",
       "\n",
       "   instrumentalness  liveness  loudness  speechiness    tempo  time_signature  \\\n",
       "0               0.0    0.1210    -6.462       0.0251   76.009               4   \n",
       "1               0.0    0.0962    -2.098       0.0323  105.586               4   \n",
       "2               0.0    0.1190    -6.941       0.0231   99.953               4   \n",
       "3               0.0    0.3200    -2.881       0.0324  115.028               4   \n",
       "4               0.0    0.1230    -5.769       0.0266  175.558               4   \n",
       "\n",
       "   valence  \n",
       "0    0.425  \n",
       "1    0.821  \n",
       "2    0.289  \n",
       "3    0.428  \n",
       "4    0.261  "
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# View data\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 247 entries, 0 to 246\n",
      "Data columns (total 16 columns):\n",
      " #   Column            Non-Null Count  Dtype  \n",
      "---  ------            --------------  -----  \n",
      " 0   name              247 non-null    object \n",
      " 1   album             247 non-null    object \n",
      " 2   artist            247 non-null    object \n",
      " 3   release_date      247 non-null    object \n",
      " 4   length            247 non-null    int64  \n",
      " 5   popularity        247 non-null    int64  \n",
      " 6   danceability      247 non-null    float64\n",
      " 7   acousticness      247 non-null    float64\n",
      " 8   energy            247 non-null    float64\n",
      " 9   instrumentalness  247 non-null    float64\n",
      " 10  liveness          247 non-null    float64\n",
      " 11  loudness          247 non-null    float64\n",
      " 12  speechiness       247 non-null    float64\n",
      " 13  tempo             247 non-null    float64\n",
      " 14  time_signature    247 non-null    int64  \n",
      " 15  valence           247 non-null    float64\n",
      "dtypes: float64(9), int64(3), object(4)\n",
      "memory usage: 31.0+ KB\n"
     ]
    }
   ],
   "source": [
    "# View structure of the data\n",
    "df.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Check for missing values\n",
    "df.isna().values.any()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.2 Cleaning the Dataset\n",
    "\n",
    "I performed some data cleaning on the datasets including:\n",
    "\n",
    "- Replacing the apostrophes for consistency and facilitate future string searches\n",
    "- Grouping tracks to albums to match the tracklisting of Taylor's Version (e.g., *Ronan* is included on *Red (Taylor's Version)* but not in the original)\n",
    "- Renaming album titles for readability"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Replace apostrophes for consistency and facilitate string searches\n",
    "df = df.replace(\"’\", \"'\", regex = True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Group Fearless tracks\n",
    "df['album'] = df['album'].replace(['Fearless Platinum Edition',\n",
    "                                   'Today Was A Fairytale'], 'Fearless')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Group Red tracks\n",
    "df['album'] = df['album'].replace(['Red (Deluxe Edition)',\n",
    "                                   'Ronan',\n",
    "                                   'The Breaker',\n",
    "                                   'Babe (feat. Taylor Swift)'], 'Red')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Group miscellaneous tracks\n",
    "df['album'] = df['album'].replace(['Sweeter Than Fiction',\n",
    "                                  'The Hunger Games: Songs From District 12 And Beyond',\n",
    "                                  'Only The Young (Featured in Miss Americana)',\n",
    "                                  'Beautiful Ghosts (From The Motion Picture \"Cats\")',\n",
    "                                  'Christmas Tree Farm'], 'Other')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Rename album titles for readability\n",
    "df['album'] = df['album'].replace(['Speak Now (Deluxe Edition)'], 'Speak Now')\n",
    "df['album'] = df['album'].replace(['1989 (Deluxe Edition)'], '1989')\n",
    "df['album'] = df['album'].replace(['folklore (deluxe version)'], 'folklore')\n",
    "df['album'] = df['album'].replace(['evermore (deluxe version)'], 'evermore')\n",
    "df['album'] = df['album'].replace(['Midnights (3am Edition)'], 'Midnights')\n",
    "df['album'] = df['album'].replace(['This Love (Taylor\\'s Version)'], '1989 (Taylor\\'s Version)')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.3 Exporting the Final Dataset\n",
    "\n",
    "Finally, I exported the dataset to the csv file *swift_data.csv*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.to_csv('data/swift_data.csv', sep = ',', index = False)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}