code/01_data_collection_and_cleaning.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](../images/bunker_sontec.jpeg)\n",
    "<br>\n",
    "Photo: [*The Bunker*](https://www.thebunkerstudio.com/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# The Sound of Science\n",
    "### Notebook 1: Data Collection & Cleaning\n",
    "#### Adam Zucker\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The Problem\n",
    "\n",
    "The advent of affordable tools for home recording over the last 20 years has made music creation more accessible than ever before. Simultaneously, social media platforms such as Instagram and Facebook have made it similarly cheap and easy to advertise yourself to a targeted audience. However, most budding musicians, and even seasoned industry veterans, don't always know the best way to reach new fans; the *how*, *who*, and *where* questions always loom tall when it's time for a new release.\n",
    "\n",
    "**In this study, I intend to create a recommender system to show similarities between the music of a new artist and that of popular artists around the globe in order to generate basic recommendations to best connect new artists with new fans. I'll base this model on data from hundreds of thousands of songs hosted on the popular streaming platform Spotify, along with metrics I'll extract from a new artist's music using a Python library called Librosa.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Background\n",
    "\n",
    "With over 15 years of experience in the music industry, first as a musician and performer, and later as a sound designer, audio engineer, and producer, I believe I'm in a unique position to address this problem. I have an intimate knowledge of the process of creating music and much of the theory behind it on both sides of the mixing board, as well as the struggles of finding and connecting with your target audience as a new artist. You can find more information about my history in the industry on [LinkedIn](https://www.linkedin.com/in/awzucker/), and samples of my recent work as part of a collaborative project called [NO EVENS](https://noevens.com/).\n",
    "\n",
    "I'm also fortunate enough to have many long-time friends in the industry, two of whom I spoke to regarding this project: [Jacob Bergson](https://www.tautmusic.com/), an independent musician and audio engineer at [The Bunker](https://www.thebunkerstudio.com/) recording studio in Brooklyn, NY, and [Joseph Kelley](https://www.linkedin.com/in/jjkelley/), the former Executive Director of Artist Relations at [Billboard](https://www.billboard.com/). Jacob was able to provide me with excellent insight on his extensive experience engineering multiple genres of music, which helped immensely as I decided how to engineer my own metrics to represent some of the more intangible qualities of music. Joe, on the other hand, has a wealth of knowledge to offer on the business landscape of the music industry: where and among what demographics artists are trending, what makes artists stand out in a digital landscape where anyone can easily post their music online, and so on."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Scope\n",
    "\n",
    "This project is a product of my immense passion for music and the process of creating and capturing it for posterity. The ultimate vision for this project is to create a user-facing app allowing anyone and everyone to upload their music, and in return, get a list of artists and songs that their music most closely resembles. From there, I intend to incorporate global listening trend data - where and to what demographic in each geographical region those similar artists are generating large numbers of streams, sales, and mentions on social media. This would ideally take the form of an interactive map, where the artist could see density and distribution of potential fans around the globe, with more granular demographic and trend data available at a click.\n",
    "\n",
    "Due to data accessibility and time constraints, the following notebooks contain the first iteration of the recommender system to determine similarity between an \"unheard\" song and roughly 60,000 songs collected from Spotify (via [Kaggle](https://www.kaggle.com/yamaerenay/spotify-dataset-19212020-160k-tracks?select=data.csv)). I certainly hope it gives you some small picture as to what I hope to achieve!\n",
    "\n",
    "Throughout this project, I also learned that quite a bit of the currently available audio data, in the form of statistical metrics, is flawed to some extent. This is, of course only in my opinion, which is precisely the point - so many qualities of how humans perceive and emotionally react to music are subjective. One song could be in the key of A minor, have a tempo of 107 beats per minute, a 4/4 time signature, an RMS loudness of -8dBF, the same instrumental-to-vocal ratio, and still sound nothing like another song with those exact same metrics. Through my exploration of Spotify's data and the metrics I generated with the help of the Python library [Librosa](https://librosa.org/), I've formed the opinion that, while some of music's intangibles will forever remain intangible, a careful and collective consideration of the intersections the ways in which computers experience music and how that relates to the human experiene of it has amazing potential to help connect artists with fans."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Contents\n",
    "- **Section 1:** Package and data imports\n",
    "- **Section 2:** Data cleaning, feature selection, and initial feature engineering\n",
    "- **Section 3:** Data exports"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "---\n",
    "### Section 1\n",
    "#### Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "import scipy as sp\n",
    "\n",
    "import os\n",
    "import IPython.display as ipd\n",
    "\n",
    "import librosa as lib"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**BELOW:** Importing master Spotify song dataset, sourced from [Kaggle](https://www.kaggle.com/yamaerenay/spotify-dataset-19212020-160k-tracks?select=data.csv). Some brief descriptions of less tangible features, as defined by [Spotify](https://developer.spotify.com/documentation/web-api/reference/):\n",
    "- **Acousticness:** A confidence measure from 0.0 to 1.0 of whether the track is acoustic.\n",
    "- **Danceability:** How suitable a track is for dancing based on a combination of musical elements including tempo, rhythm stability, beat strength, and overall regularity. A value of 0.0 is least danceable and 1.0 is most danceable. \n",
    "- **Energy:** A measure from 0.0 to 1.0 and represents a perceptual measure of intensity and activity. Typically, energetic tracks feel fast, loud, and noisy. For example, death metal has high energy, while a Bach prelude scores low on the scale. Perceptual features contributing to this attribute include dynamic range, perceived loudness, timbre, onset rate, and general entropy.\n",
    "- **Instrumentalness:** Predicts whether a track contains vocals. “Ooh” and “aah” sounds are treated as instrumental in this context. Rap or spoken word tracks are clearly “vocal”. The closer the instrumentalness value is to 1.0, the greater likelihood the track contains no vocal content. Values above 0.5 are intended to represent instrumental tracks, but confidence is higher as the value approaches 1.0.\n",
    "- **Liveness:** Detects the presence of an audience in the recording. Higher liveness values represent an increased probability that the track was performed live. A value above 0.8 provides strong likelihood that the track is live.\n",
    "- **Loudness:** The overall loudness of a track in decibels (dB). Loudness values are averaged across the entire track and are useful for comparing relative loudness of tracks. Loudness is the quality of a sound that is the primary psychological correlate of physical strength (amplitude). Values typically range between -60 and 0 dB.\n",
    "- **Popularity:** The popularity of a track is a value between 0 and 100, with 100 being the most popular. The popularity is calculated by algorithm and is based, in the most part, on the total number of plays the track has had and how recent those plays are. Generally speaking, songs that are being played a lot now will have a higher popularity than songs that were played a lot in the past. Duplicate tracks (e.g., the same track from a single and an album) are rated independently. Artist and album popularity is derived mathematically from track popularity.\n",
    "- **Speechiness:** Detects the presence of spoken words in a track. The more exclusively speech-like the recording (e.g., talk show, audio book, poetry), the closer to 1.0 the attribute value. Values above 0.66 describe tracks that are probably made entirely of spoken words. Values between 0.33 and 0.66 describe tracks that may contain both music and speech, either in sections or layered, including such cases as rap music. Values below 0.33 most likely represent music and other non-speech-like tracks.\n",
    "- **Valence:** A measure from 0.0 to 1.0 describing the musical positiveness conveyed by a track. Tracks with high valence sound more positive (e.g., happy, cheerful, euphoric), while tracks with low valence sound more negative (e.g., sad, depressed, angry)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>valence</th>\n",
       "      <th>year</th>\n",
       "      <th>acousticness</th>\n",
       "      <th>artists</th>\n",
       "      <th>danceability</th>\n",
       "      <th>duration_ms</th>\n",
       "      <th>energy</th>\n",
       "      <th>explicit</th>\n",
       "      <th>id</th>\n",
       "      <th>instrumentalness</th>\n",
       "      <th>key</th>\n",
       "      <th>liveness</th>\n",
       "      <th>loudness</th>\n",
       "      <th>mode</th>\n",
       "      <th>name</th>\n",
       "      <th>popularity</th>\n",
       "      <th>release_date</th>\n",
       "      <th>speechiness</th>\n",
       "      <th>tempo</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.0594</td>\n",
       "      <td>1921</td>\n",
       "      <td>0.982</td>\n",
       "      <td>['Sergei Rachmaninoff', 'James Levine', 'Berli...</td>\n",
       "      <td>0.279</td>\n",
       "      <td>831667</td>\n",
       "      <td>0.211</td>\n",
       "      <td>0</td>\n",
       "      <td>4BJqT0PrAfrxzMOxytFOIz</td>\n",
       "      <td>0.878000</td>\n",
       "      <td>10</td>\n",
       "      <td>0.665</td>\n",
       "      <td>-20.096</td>\n",
       "      <td>1</td>\n",
       "      <td>Piano Concerto No. 3 in D Minor, Op. 30: III. ...</td>\n",
       "      <td>4</td>\n",
       "      <td>1921</td>\n",
       "      <td>0.0366</td>\n",
       "      <td>80.954</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.9630</td>\n",
       "      <td>1921</td>\n",
       "      <td>0.732</td>\n",
       "      <td>['Dennis Day']</td>\n",
       "      <td>0.819</td>\n",
       "      <td>180533</td>\n",
       "      <td>0.341</td>\n",
       "      <td>0</td>\n",
       "      <td>7xPhfUan2yNtyFG0cUWkt8</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>7</td>\n",
       "      <td>0.160</td>\n",
       "      <td>-12.441</td>\n",
       "      <td>1</td>\n",
       "      <td>Clancy Lowered the Boom</td>\n",
       "      <td>5</td>\n",
       "      <td>1921</td>\n",
       "      <td>0.4150</td>\n",
       "      <td>60.936</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.0394</td>\n",
       "      <td>1921</td>\n",
       "      <td>0.961</td>\n",
       "      <td>['KHP Kridhamardawa Karaton Ngayogyakarta Hadi...</td>\n",
       "      <td>0.328</td>\n",
       "      <td>500062</td>\n",
       "      <td>0.166</td>\n",
       "      <td>0</td>\n",
       "      <td>1o6I8BglA6ylDMrIELygv1</td>\n",
       "      <td>0.913000</td>\n",
       "      <td>3</td>\n",
       "      <td>0.101</td>\n",
       "      <td>-14.850</td>\n",
       "      <td>1</td>\n",
       "      <td>Gati Bali</td>\n",
       "      <td>5</td>\n",
       "      <td>1921</td>\n",
       "      <td>0.0339</td>\n",
       "      <td>110.339</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.1650</td>\n",
       "      <td>1921</td>\n",
       "      <td>0.967</td>\n",
       "      <td>['Frank Parker']</td>\n",
       "      <td>0.275</td>\n",
       "      <td>210000</td>\n",
       "      <td>0.309</td>\n",
       "      <td>0</td>\n",
       "      <td>3ftBPsC5vPBKxYSee08FDH</td>\n",
       "      <td>0.000028</td>\n",
       "      <td>5</td>\n",
       "      <td>0.381</td>\n",
       "      <td>-9.316</td>\n",
       "      <td>1</td>\n",
       "      <td>Danny Boy</td>\n",
       "      <td>3</td>\n",
       "      <td>1921</td>\n",
       "      <td>0.0354</td>\n",
       "      <td>100.109</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.2530</td>\n",
       "      <td>1921</td>\n",
       "      <td>0.957</td>\n",
       "      <td>['Phil Regan']</td>\n",
       "      <td>0.418</td>\n",
       "      <td>166693</td>\n",
       "      <td>0.193</td>\n",
       "      <td>0</td>\n",
       "      <td>4d6HGyGT8e121BsdKmw9v6</td>\n",
       "      <td>0.000002</td>\n",
       "      <td>3</td>\n",
       "      <td>0.229</td>\n",
       "      <td>-10.096</td>\n",
       "      <td>1</td>\n",
       "      <td>When Irish Eyes Are Smiling</td>\n",
       "      <td>2</td>\n",
       "      <td>1921</td>\n",
       "      <td>0.0380</td>\n",
       "      <td>101.665</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   valence  year  acousticness  \\\n",
       "0   0.0594  1921         0.982   \n",
       "1   0.9630  1921         0.732   \n",
       "2   0.0394  1921         0.961   \n",
       "3   0.1650  1921         0.967   \n",
       "4   0.2530  1921         0.957   \n",
       "\n",
       "                                             artists  danceability  \\\n",
       "0  ['Sergei Rachmaninoff', 'James Levine', 'Berli...         0.279   \n",
       "1                                     ['Dennis Day']         0.819   \n",
       "2  ['KHP Kridhamardawa Karaton Ngayogyakarta Hadi...         0.328   \n",
       "3                                   ['Frank Parker']         0.275   \n",
       "4                                     ['Phil Regan']         0.418   \n",
       "\n",
       "   duration_ms  energy  explicit                      id  instrumentalness  \\\n",
       "0       831667   0.211         0  4BJqT0PrAfrxzMOxytFOIz          0.878000   \n",
       "1       180533   0.341         0  7xPhfUan2yNtyFG0cUWkt8          0.000000   \n",
       "2       500062   0.166         0  1o6I8BglA6ylDMrIELygv1          0.913000   \n",
       "3       210000   0.309         0  3ftBPsC5vPBKxYSee08FDH          0.000028   \n",
       "4       166693   0.193         0  4d6HGyGT8e121BsdKmw9v6          0.000002   \n",
       "\n",
       "   key  liveness  loudness  mode  \\\n",
       "0   10     0.665   -20.096     1   \n",
       "1    7     0.160   -12.441     1   \n",
       "2    3     0.101   -14.850     1   \n",
       "3    5     0.381    -9.316     1   \n",
       "4    3     0.229   -10.096     1   \n",
       "\n",
       "                                                name  popularity release_date  \\\n",
       "0  Piano Concerto No. 3 in D Minor, Op. 30: III. ...           4         1921   \n",
       "1                            Clancy Lowered the Boom           5         1921   \n",
       "2                                          Gati Bali           5         1921   \n",
       "3                                          Danny Boy           3         1921   \n",
       "4                        When Irish Eyes Are Smiling           2         1921   \n",
       "\n",
       "   speechiness    tempo  \n",
       "0       0.0366   80.954  \n",
       "1       0.4150   60.936  \n",
       "2       0.0339  110.339  \n",
       "3       0.0354  100.109  \n",
       "4       0.0380  101.665  "
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_csv('../data_raw/kg_spotify_master_data.csv')\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(170653, 19)"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 170653 entries, 0 to 170652\n",
      "Data columns (total 19 columns):\n",
      " #   Column            Non-Null Count   Dtype  \n",
      "---  ------            --------------   -----  \n",
      " 0   valence           170653 non-null  float64\n",
      " 1   year              170653 non-null  int64  \n",
      " 2   acousticness      170653 non-null  float64\n",
      " 3   artists           170653 non-null  object \n",
      " 4   danceability      170653 non-null  float64\n",
      " 5   duration_ms       170653 non-null  int64  \n",
      " 6   energy            170653 non-null  float64\n",
      " 7   explicit          170653 non-null  int64  \n",
      " 8   id                170653 non-null  object \n",
      " 9   instrumentalness  170653 non-null  float64\n",
      " 10  key               170653 non-null  int64  \n",
      " 11  liveness          170653 non-null  float64\n",
      " 12  loudness          170653 non-null  float64\n",
      " 13  mode              170653 non-null  int64  \n",
      " 14  name              170653 non-null  object \n",
      " 15  popularity        170653 non-null  int64  \n",
      " 16  release_date      170653 non-null  object \n",
      " 17  speechiness       170653 non-null  float64\n",
      " 18  tempo             170653 non-null  float64\n",
      "dtypes: float64(9), int64(6), object(4)\n",
      "memory usage: 24.7+ MB\n"
     ]
    }
   ],
   "source": [
    "df.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "---\n",
    "### Section 2\n",
    "#### Data Cleaning & Feature Selection"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Dropping the 'explicit' column, as this won't be relevant for audio analysis\n",
    "df.drop(columns='explicit', inplace=True)\n",
    "\n",
    "# According to Spotify, 'liveness' predicts whether or not the track was recorded as part of a live show.\n",
    "# Spotify's algorithm looks for audience noise in the track to make these predictions. A value > 0.8 suggests the track is a live recording.\n",
    "# Here, isolating and dropping assumed \"live\" recordings, as their sonic qualities usually differ quite a bit from studio-recorded music.\n",
    "df = df[(df['liveness'] <= 0.8)]\n",
    "\n",
    "# After checking the resulting dataframe, some live recording results still appeared. Dropping them here.\n",
    "df = df[(df['name'].str.contains('- live', case=False)) == False]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(165185, 18)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['valence', 'year', 'acousticness', 'artists', 'danceability',\n",
       "       'duration_ms', 'energy', 'id', 'instrumentalness', 'key', 'liveness',\n",
       "       'loudness', 'mode', 'name', 'popularity', 'release_date', 'speechiness',\n",
       "       'tempo'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**BELOW:** Some songs have duplicate entries in the dataframe. Some reasons duplicates can legitimately occur include cover versions by other artists, *single* and *album* versions of the same song, different songs with the same name, and often with orchestral music, different recordings of the same piece by different ensembles.\n",
    "\n",
    "However, I want to be sure not to include any duplicate songs that actually have identical data with respect to musical features. Below, I'm using a simple `drop_duplicates()` method, factoring out the song's `id` (this is always unique on Spotify) and `popularity`. I'll also investigate the other potentially problematic duplicate case, in which there are album and single versions of a song that are very similar, but don't have identical metrics. This can happen because of the way albums are formatted versus the way singles are formatted."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Defining the subset of columns I'll check for duplicate entries when passing rows into drop_duplicates()\n",
    "dd_list = ['name', 'artists', 'tempo', 'key', 'loudness', 'mode', 'duration_ms', 'energy', 'instrumentalness', 'speechiness', \n",
    "           'acousticness', 'danceability', 'valence', 'liveness', 'year', 'release_date']\n",
    "\n",
    "df.drop_duplicates(subset=dd_list, inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Also dropping 'release_date' column since I already have 'year'. For the scope of this project, an exact release date isn't necessary.\n",
    "df.drop(columns='release_date', inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(163256, 17)"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**BELOW**: Rounding `tempo` values to the nearest single decimal place. A few notes on this choice, and on tempo in general:\n",
    "1. `tempo` is measured in beats per minute, and is generally expressed in integer values in the music world, though decimals can exist. During my time in the industry, I've never seen tempo expressed beyond one decimal place.\n",
    "2. The decimal places exist in the original dataframe because the `tempo` metric is averaged for the entire song. Many songs recorded before the advent of digital recording and *click tracks* (a regular, metronomic pulse musicians hear in their headphones when recording to keep a steady pace) do not have a consistent tempo, due to natural human error in timing. \n",
    "3. Some old songs have also been altered by the idiosyncrasies of analog recording - an improperly aligned tape machine can cause small tempo fluctuations on playback, or when transferred to a new tape reel during the mastering process.\n",
    "4. Ambient, orchestral, purely acoustic, purely vocal, and other such pieces without a strong pulse can also \"trick\" Spotify's tempo algorithm. Computers are largely able to infer tempo based on regular pulses (often in the attack sounds of drums and other rhythm instruments, known as *transients*) that machine learning algorithms can use as something like anchor points. Measuring the temporal distance between these \"anchors\" ideally yields a tempo."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Rounding \"tempo\" to a single decimal place\n",
    "df['tempo'] = round(df['tempo'], 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "120.0    909\n",
       "128.0    694\n",
       "130.0    692\n",
       "100.0    671\n",
       "140.0    643\n",
       "        ... \n",
       "34.5       1\n",
       "36.6       1\n",
       "195.5      1\n",
       "32.8       1\n",
       "215.8      1\n",
       "Name: tempo, Length: 1774, dtype: int64"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['tempo'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "count    163256.000000\n",
       "mean        116.805682\n",
       "std          30.695406\n",
       "min           0.000000\n",
       "25%          93.400000\n",
       "50%         114.700000\n",
       "75%         135.400000\n",
       "max         243.500000\n",
       "Name: tempo, dtype: float64"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# I already see some oddities with tempo - for example, the minimum tempo in the dataframe is 0, which isn't possible.\n",
    "df['tempo'].describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Tempo** cannot be 0, or no beats per minute. In fact, barely any music, and especially modern music, exists with a tempo of less than 40 beats per minute. As you can see above, the bottom 25th percentile of tempos fall below 96 bpm, with a mean of around 120 bpm. This in mind, I'm dropping all entries with a tempo below 40 bpm, as I don't believe those measurements to be reliable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Dropping tracks with a tempo less than 40. Most of these appear to be ambient sounds anyway, like \"Fan Sounds\" and \"White Noise Meditation\"\n",
    "df = df[df['tempo'] >= 40]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As mentioned **above**, age can drastically affect an audio recording. In the early half of the 20th century, recording technology was still very much in its infancy. The full frequency range of human hearing (20Hz to 20kHz) wasn't reproduceable with the microphones and media of the time, and additional noise, often in the form of white noise, mechanical noise, or electrical hum, was clearly audible. With this in mind, I'm going to drop entries for songs recorded prior to 1960, as noise and *generation loss* become larger problems as you go back further in time. The excessive noise and poor sound reproduction of many of these recordings are likely to skew their metrics."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Dropping all tracks from before 1960\n",
    "df = df[(df['year'] >= 1960)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2018    2080\n",
       "1967    1964\n",
       "1998    1959\n",
       "2001    1957\n",
       "2014    1955\n",
       "        ... \n",
       "1977    1782\n",
       "1960    1772\n",
       "2016    1771\n",
       "1976    1769\n",
       "1961    1759\n",
       "Name: year, Length: 61, dtype: int64"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['year'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "count    115200.000000\n",
       "mean       1990.298620\n",
       "std          17.578021\n",
       "min        1960.000000\n",
       "25%        1975.000000\n",
       "50%        1990.000000\n",
       "75%        2005.000000\n",
       "max        2020.000000\n",
       "Name: year, dtype: float64"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['year'].describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**ABOVE:** Looking at the distribution of `year` in the dataframe, it appears that the top 50th percentile of songs (based on year) in the dataframe are from 1990 or later.\n",
    "\n",
    "**BELOW:** Based on the metrics above, I'm going to do some EDA regarding `year` and `popularity`. The goal is to determine if it's wise to drop the bottom 50th percentile of years, as Spotify's popularity algorithm factors a song's newness into its popularity score. Looking at popularity statistics by year reinforces this weighting, as mean popularity almost universally increases year over year."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "count    115200.000000\n",
       "mean         42.542266\n",
       "std          15.506639\n",
       "min           0.000000\n",
       "25%          31.000000\n",
       "50%          42.000000\n",
       "75%          53.000000\n",
       "max         100.000000\n",
       "Name: popularity, dtype: float64"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['popularity'].describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>count</th>\n",
       "      <th>mean</th>\n",
       "      <th>std</th>\n",
       "      <th>min</th>\n",
       "      <th>25%</th>\n",
       "      <th>50%</th>\n",
       "      <th>75%</th>\n",
       "      <th>max</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>year</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1960</th>\n",
       "      <td>1772.0</td>\n",
       "      <td>19.815463</td>\n",
       "      <td>11.082582</td>\n",
       "      <td>5.0</td>\n",
       "      <td>12.0</td>\n",
       "      <td>17.0</td>\n",
       "      <td>25.0</td>\n",
       "      <td>76.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1961</th>\n",
       "      <td>1759.0</td>\n",
       "      <td>19.774872</td>\n",
       "      <td>11.254397</td>\n",
       "      <td>4.0</td>\n",
       "      <td>12.0</td>\n",
       "      <td>17.0</td>\n",
       "      <td>25.0</td>\n",
       "      <td>78.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1962</th>\n",
       "      <td>1820.0</td>\n",
       "      <td>22.325275</td>\n",
       "      <td>11.953213</td>\n",
       "      <td>6.0</td>\n",
       "      <td>13.0</td>\n",
       "      <td>19.0</td>\n",
       "      <td>28.0</td>\n",
       "      <td>78.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1963</th>\n",
       "      <td>1910.0</td>\n",
       "      <td>25.761780</td>\n",
       "      <td>11.601255</td>\n",
       "      <td>9.0</td>\n",
       "      <td>17.0</td>\n",
       "      <td>23.0</td>\n",
       "      <td>31.0</td>\n",
       "      <td>83.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1964</th>\n",
       "      <td>1860.0</td>\n",
       "      <td>26.420968</td>\n",
       "      <td>11.903986</td>\n",
       "      <td>9.0</td>\n",
       "      <td>18.0</td>\n",
       "      <td>24.0</td>\n",
       "      <td>32.0</td>\n",
       "      <td>85.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2016</th>\n",
       "      <td>1771.0</td>\n",
       "      <td>59.659514</td>\n",
       "      <td>11.760311</td>\n",
       "      <td>0.0</td>\n",
       "      <td>56.0</td>\n",
       "      <td>60.0</td>\n",
       "      <td>65.0</td>\n",
       "      <td>88.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2017</th>\n",
       "      <td>1926.0</td>\n",
       "      <td>63.237799</td>\n",
       "      <td>11.311437</td>\n",
       "      <td>0.0</td>\n",
       "      <td>59.0</td>\n",
       "      <td>63.0</td>\n",
       "      <td>68.0</td>\n",
       "      <td>87.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2018</th>\n",
       "      <td>2080.0</td>\n",
       "      <td>63.375481</td>\n",
       "      <td>15.412989</td>\n",
       "      <td>0.0</td>\n",
       "      <td>61.0</td>\n",
       "      <td>65.0</td>\n",
       "      <td>70.0</td>\n",
       "      <td>89.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019</th>\n",
       "      <td>1926.0</td>\n",
       "      <td>65.407061</td>\n",
       "      <td>16.018737</td>\n",
       "      <td>0.0</td>\n",
       "      <td>63.0</td>\n",
       "      <td>67.0</td>\n",
       "      <td>73.0</td>\n",
       "      <td>94.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2020</th>\n",
       "      <td>1922.0</td>\n",
       "      <td>66.724766</td>\n",
       "      <td>19.135880</td>\n",
       "      <td>0.0</td>\n",
       "      <td>66.0</td>\n",
       "      <td>71.0</td>\n",
       "      <td>76.0</td>\n",
       "      <td>100.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>61 rows × 8 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       count       mean        std  min   25%   50%   75%    max\n",
       "year                                                            \n",
       "1960  1772.0  19.815463  11.082582  5.0  12.0  17.0  25.0   76.0\n",
       "1961  1759.0  19.774872  11.254397  4.0  12.0  17.0  25.0   78.0\n",
       "1962  1820.0  22.325275  11.953213  6.0  13.0  19.0  28.0   78.0\n",
       "1963  1910.0  25.761780  11.601255  9.0  17.0  23.0  31.0   83.0\n",
       "1964  1860.0  26.420968  11.903986  9.0  18.0  24.0  32.0   85.0\n",
       "...      ...        ...        ...  ...   ...   ...   ...    ...\n",
       "2016  1771.0  59.659514  11.760311  0.0  56.0  60.0  65.0   88.0\n",
       "2017  1926.0  63.237799  11.311437  0.0  59.0  63.0  68.0   87.0\n",
       "2018  2080.0  63.375481  15.412989  0.0  61.0  65.0  70.0   89.0\n",
       "2019  1926.0  65.407061  16.018737  0.0  63.0  67.0  73.0   94.0\n",
       "2020  1922.0  66.724766  19.135880  0.0  66.0  71.0  76.0  100.0\n",
       "\n",
       "[61 rows x 8 columns]"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['popularity'].groupby(by=df['year']).describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Based on the above popularity scores, I'm dropping all entries prior to 1990.\n",
    "df = df[(df['year'] >= 1990)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(59526, 17)"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**BELOW:** In music theory, we often talk about *modality*, in addition to *key*, to quantify note relationships within the context of a piece of music. \n",
    "\n",
    "In modern theory, there are seven modes. Each mode has a Greek name and a Roman numeral associated with it that represents the *scale degree* on which the mode starts. If the Roman numeral is uppercase, the mode is considered to have a *major* tonality, while if the numeral is lowercase, the tonality is considered *minor* (there's one exception to this: the Locrian mode, which has a *diminished* tonality, but could also be considered a *minor* variant). In order, the modes are Ionian ($I$), Dorian ($ii$), Phrygian ($iii$), Lydian ($IV$), Mixolydian ($V$), Aeolian ($vi$), and Locrian ($vii^ø$). When a song key is given in *major*, it's referring to the Ionian mode, where the key's *tonic*, or first note/fundamental pitch is the first note in the Ionian scale. Each *major key* has a direct relationship with a specific *minor key*, called the *relative minor*, which is given by the Aeolian mode (also known as the *natural minor scale*) of the corresponding *major* key. The relative *natural minor scale* to a *major key* begins three *semitones* below the *major scale's tonic* (e.g., C Major contains the same notes as A minor, G Major as E minor, F Major as D minor, etc.).\n",
    "\n",
    "> As you'll see in the chart below (in the `Notation On White Notes` column), each successive mode contains all the same notes as the relative Ionian scale, but simply starts on the next *scale degree*, which also corresponds to the Roman numeral. So, in this example, one could say:\n",
    ">\n",
    "> **C Ionian = D Dorian = E Phrygian = F Lydian = G Mixolydian = A Aeolian = B Locrian**\n",
    ">\n",
    "> Though each scale will sound different and elicit different emotional responses because of the intervallic shifts between consecutive notes, all are comprised of the same seven notes: **C, D, E, F, G, A, B**.\n",
    ">\n",
    "> <img src='../images/modes.png' alt='drawing' width='900'/>\n",
    "\n",
    "This is important in my study because, as humans, we can generally distinguish between *major* and *minor* keys by ear. *Major* keys tend to have a \"happy\" or \"positive\" sound, while *minor* keys tend to have a \"darker\" or \"sadder\" sound. However, computers don't always get this classification right, due to the fact that the notes are the same in corresponding *major* and *minor* keys, and the emotions each sequence of the same notes elicits is hard to quantify statistically. Therefore, if Spotify or Librosa classify a song as being in A minor, for example, I'd like my algorithm to also factor in the possibility that the key could in fact be represented as C Major too, so as not to rule out any similar tracks based on a misclassified key.\n",
    "\n",
    "Spotify only offers values of `0` or `1` for `mode`, where `0` represents a *minor* key (or the Aeolian mode), and `1` represents a *major* key (or the Ionian mode)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>valence</th>\n",
       "      <th>year</th>\n",
       "      <th>acousticness</th>\n",
       "      <th>artists</th>\n",
       "      <th>danceability</th>\n",
       "      <th>duration_ms</th>\n",
       "      <th>energy</th>\n",
       "      <th>id</th>\n",
       "      <th>instrumentalness</th>\n",
       "      <th>key</th>\n",
       "      <th>liveness</th>\n",
       "      <th>loudness</th>\n",
       "      <th>mode</th>\n",
       "      <th>name</th>\n",
       "      <th>popularity</th>\n",
       "      <th>speechiness</th>\n",
       "      <th>tempo</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Empty DataFrame\n",
       "Columns: [valence, year, acousticness, artists, danceability, duration_ms, energy, id, instrumentalness, key, liveness, loudness, mode, name, popularity, speechiness, tempo]\n",
       "Index: []"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Modality (here defined as 0 for a minor key, 1 for a major key)\n",
    "# Checking for entries where \"mode\" is equal to -1. This is the equivalent of a null or non-conclusive analysis.\n",
    "# None found\n",
    "df[df['mode'] == -1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>valence</th>\n",
       "      <th>year</th>\n",
       "      <th>acousticness</th>\n",
       "      <th>artists</th>\n",
       "      <th>danceability</th>\n",
       "      <th>duration_ms</th>\n",
       "      <th>energy</th>\n",
       "      <th>id</th>\n",
       "      <th>instrumentalness</th>\n",
       "      <th>key</th>\n",
       "      <th>liveness</th>\n",
       "      <th>loudness</th>\n",
       "      <th>mode</th>\n",
       "      <th>name</th>\n",
       "      <th>popularity</th>\n",
       "      <th>speechiness</th>\n",
       "      <th>tempo</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Empty DataFrame\n",
       "Columns: [valence, year, acousticness, artists, danceability, duration_ms, energy, id, instrumentalness, key, liveness, loudness, mode, name, popularity, speechiness, tempo]\n",
       "Index: []"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Key (here defined as 0-11, where 0 represents C, and keys increase chromatically with integer)\n",
    "# Checking for entries where \"key\" is equal to -1. This is the equivalent of a null or non-conclusive analysis.\n",
    "# None found\n",
    "df[df['key'] == -1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**BELOW:** Incorporating the relative major or minor key into my dataframe by engineering `secondary_key` and `secondary_mode`. If the original mode is major (`mode == 1`), then I put `0`, or the number representing a minor modality in the corresponding `secondary_mode`, and vice versa.\n",
    "\n",
    "Once I compute these secondary modes, I can compute the relative major or minor key by subtracting three semitones if the original key was major, and adding three semitones if the original key was minor."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Creating secondary key and mode columns for my dataframe\n",
    "df['secondary_key'] = 0\n",
    "df['secondary_mode'] = 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Inverting the original mode to fill the 'secondary_mode' column\n",
    "df.loc[(df['mode'] == 0, 'secondary_mode')] = 1\n",
    "df.loc[(df['mode'] == 1, 'secondary_mode')] = 0\n",
    "\n",
    "# Adding/subtracting 3 semitones from the original key to fill the 'secondary_key' column\n",
    "df.loc[(df['secondary_mode'] == 0, 'secondary_key')] = df['key'] - 3\n",
    "df.loc[(df['secondary_mode'] == 1, 'secondary_key')] = df['key'] + 3\n",
    "\n",
    "# There are 12 keys total, represented as numbers 0 to 11. If the math above results in a number greater than 11, \n",
    "# I'll have to subtract 12 semitones, or one full octave, to get the appropriate key value. The inverse is true\n",
    "# if the result is less than 0.\n",
    "df.loc[(df['secondary_key'] > 11, 'secondary_key')] = df['secondary_key'] - 12\n",
    "df.loc[(df['secondary_key'] < 0, 'secondary_key')] = df['secondary_key'] + 12"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(59526, 19)"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Engineering a feature for 'full_key', which contains the letter representing the key's fundamental pitch, and the modality,\n",
    "# here representing whether the key is major or minor\n",
    "df['key_letter'] = df['key'].map({0: 'C', 1: 'C#', 2: 'D', 3: 'D#', 4: 'E', 5: 'F', 6: 'F#', 7: 'G', 8: 'G#', 9: 'A', 10: 'A#', 11: 'B'})\n",
    "df['maj_min'] = df['mode'].map({0: 'minor', 1: 'major'})\n",
    "\n",
    "df['full_key'] = df['key_letter'] + ' ' + df['maj_min']\n",
    "\n",
    "# Engineering a 'full_secondary_key' column as above\n",
    "df['secondary_key_letter'] = df['secondary_key'].map({0: 'C', 1: 'C#', 2: 'D', 3: 'D#', 4: 'E', 5: 'F', 6: 'F#', 7: 'G', 8: 'G#', 9: 'A', 10: 'A#', 11: 'B'})\n",
    "df['secondary_maj_min'] = df['secondary_mode'].map({0: 'minor', 1: 'major'})\n",
    "\n",
    "df['secondary_full_key'] = df['secondary_key_letter'] + ' ' + df['secondary_maj_min']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Dropping columns engineered to make 'full_key' and 'secondary_full_key', as they're no longer necessary\n",
    "df.drop(columns=['key_letter', 'maj_min', 'secondary_key_letter', 'secondary_maj_min'], inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['valence', 'year', 'acousticness', 'artists', 'danceability',\n",
       "       'duration_ms', 'energy', 'id', 'instrumentalness', 'key', 'liveness',\n",
       "       'loudness', 'mode', 'name', 'popularity', 'speechiness', 'tempo',\n",
       "       'secondary_key', 'secondary_mode', 'full_key', 'secondary_full_key'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Reordering columns for clarity\n",
    "df = df[['name', 'artists', 'tempo', 'key', 'mode', 'full_key', 'secondary_key', 'secondary_mode', 'secondary_full_key', \n",
    "         'loudness', 'duration_ms', 'energy', 'instrumentalness', 'speechiness', 'acousticness', 'danceability', 'valence', \n",
    "         'popularity', 'liveness', 'year', 'id']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>artists</th>\n",
       "      <th>tempo</th>\n",
       "      <th>key</th>\n",
       "      <th>mode</th>\n",
       "      <th>full_key</th>\n",
       "      <th>secondary_key</th>\n",
       "      <th>secondary_mode</th>\n",
       "      <th>secondary_full_key</th>\n",
       "      <th>loudness</th>\n",
       "      <th>...</th>\n",
       "      <th>energy</th>\n",
       "      <th>instrumentalness</th>\n",
       "      <th>speechiness</th>\n",
       "      <th>acousticness</th>\n",
       "      <th>danceability</th>\n",
       "      <th>valence</th>\n",
       "      <th>popularity</th>\n",
       "      <th>liveness</th>\n",
       "      <th>year</th>\n",
       "      <th>id</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>13606</th>\n",
       "      <td>Thunderstruck</td>\n",
       "      <td>['AC/DC']</td>\n",
       "      <td>133.5</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>E major</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>C# minor</td>\n",
       "      <td>-5.175</td>\n",
       "      <td>...</td>\n",
       "      <td>0.890</td>\n",
       "      <td>0.011700</td>\n",
       "      <td>0.0364</td>\n",
       "      <td>0.000147</td>\n",
       "      <td>0.502</td>\n",
       "      <td>0.259</td>\n",
       "      <td>83</td>\n",
       "      <td>0.2170</td>\n",
       "      <td>1990</td>\n",
       "      <td>57bgtoPSgt236HzfBOd8kj</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13607</th>\n",
       "      <td>Carol of the Bells</td>\n",
       "      <td>['Mykola Dmytrovych Leontovych', 'John Williams']</td>\n",
       "      <td>46.7</td>\n",
       "      <td>8</td>\n",
       "      <td>0</td>\n",
       "      <td>G# minor</td>\n",
       "      <td>11</td>\n",
       "      <td>1</td>\n",
       "      <td>B major</td>\n",
       "      <td>-22.507</td>\n",
       "      <td>...</td>\n",
       "      <td>0.106</td>\n",
       "      <td>0.029200</td>\n",
       "      <td>0.0448</td>\n",
       "      <td>0.994000</td>\n",
       "      <td>0.418</td>\n",
       "      <td>0.800</td>\n",
       "      <td>73</td>\n",
       "      <td>0.1790</td>\n",
       "      <td>1990</td>\n",
       "      <td>4tHqQMWSqmL6YjXwsqthDI</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13608</th>\n",
       "      <td>Man in the Box</td>\n",
       "      <td>['Alice In Chains']</td>\n",
       "      <td>106.4</td>\n",
       "      <td>8</td>\n",
       "      <td>1</td>\n",
       "      <td>G# major</td>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>F minor</td>\n",
       "      <td>-7.565</td>\n",
       "      <td>...</td>\n",
       "      <td>0.791</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0453</td>\n",
       "      <td>0.000450</td>\n",
       "      <td>0.355</td>\n",
       "      <td>0.758</td>\n",
       "      <td>73</td>\n",
       "      <td>0.0969</td>\n",
       "      <td>1990</td>\n",
       "      <td>6gZVQvQZOFpzIy3HblJ20F</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13609</th>\n",
       "      <td>Don't Go Breaking My Heart</td>\n",
       "      <td>['Elton John', 'Kiki Dee']</td>\n",
       "      <td>131.2</td>\n",
       "      <td>5</td>\n",
       "      <td>1</td>\n",
       "      <td>F major</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>D minor</td>\n",
       "      <td>-7.823</td>\n",
       "      <td>...</td>\n",
       "      <td>0.863</td>\n",
       "      <td>0.000006</td>\n",
       "      <td>0.0380</td>\n",
       "      <td>0.177000</td>\n",
       "      <td>0.742</td>\n",
       "      <td>0.776</td>\n",
       "      <td>69</td>\n",
       "      <td>0.0699</td>\n",
       "      <td>1990</td>\n",
       "      <td>5pKJtX4wBeby9qIfFhyOJj</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13610</th>\n",
       "      <td>This Christmas</td>\n",
       "      <td>['Donny Hathaway']</td>\n",
       "      <td>93.0</td>\n",
       "      <td>5</td>\n",
       "      <td>1</td>\n",
       "      <td>F major</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>D minor</td>\n",
       "      <td>-9.200</td>\n",
       "      <td>...</td>\n",
       "      <td>0.645</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0376</td>\n",
       "      <td>0.292000</td>\n",
       "      <td>0.634</td>\n",
       "      <td>0.677</td>\n",
       "      <td>70</td>\n",
       "      <td>0.2460</td>\n",
       "      <td>1990</td>\n",
       "      <td>38xhBO2AKrJnjdjVnhJES6</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 21 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                             name  \\\n",
       "13606               Thunderstruck   \n",
       "13607          Carol of the Bells   \n",
       "13608              Man in the Box   \n",
       "13609  Don't Go Breaking My Heart   \n",
       "13610              This Christmas   \n",
       "\n",
       "                                                 artists  tempo  key  mode  \\\n",
       "13606                                          ['AC/DC']  133.5    4     1   \n",
       "13607  ['Mykola Dmytrovych Leontovych', 'John Williams']   46.7    8     0   \n",
       "13608                                ['Alice In Chains']  106.4    8     1   \n",
       "13609                         ['Elton John', 'Kiki Dee']  131.2    5     1   \n",
       "13610                                 ['Donny Hathaway']   93.0    5     1   \n",
       "\n",
       "       full_key  secondary_key  secondary_mode secondary_full_key  loudness  \\\n",
       "13606   E major              1               0           C# minor    -5.175   \n",
       "13607  G# minor             11               1            B major   -22.507   \n",
       "13608  G# major              5               0            F minor    -7.565   \n",
       "13609   F major              2               0            D minor    -7.823   \n",
       "13610   F major              2               0            D minor    -9.200   \n",
       "\n",
       "       ...  energy  instrumentalness  speechiness  acousticness  danceability  \\\n",
       "13606  ...   0.890          0.011700       0.0364      0.000147         0.502   \n",
       "13607  ...   0.106          0.029200       0.0448      0.994000         0.418   \n",
       "13608  ...   0.791          0.000000       0.0453      0.000450         0.355   \n",
       "13609  ...   0.863          0.000006       0.0380      0.177000         0.742   \n",
       "13610  ...   0.645          0.000000       0.0376      0.292000         0.634   \n",
       "\n",
       "       valence  popularity  liveness  year                      id  \n",
       "13606    0.259          83    0.2170  1990  57bgtoPSgt236HzfBOd8kj  \n",
       "13607    0.800          73    0.1790  1990  4tHqQMWSqmL6YjXwsqthDI  \n",
       "13608    0.758          73    0.0969  1990  6gZVQvQZOFpzIy3HblJ20F  \n",
       "13609    0.776          69    0.0699  1990  5pKJtX4wBeby9qIfFhyOJj  \n",
       "13610    0.677          70    0.2460  1990  38xhBO2AKrJnjdjVnhJES6  \n",
       "\n",
       "[5 rows x 21 columns]"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>popularity</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>59526.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>51.144239</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>12.885932</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>43.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>51.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>60.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>100.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         popularity\n",
       "count  59526.000000\n",
       "mean      51.144239\n",
       "std       12.885932\n",
       "min        0.000000\n",
       "25%       43.000000\n",
       "50%       51.000000\n",
       "75%       60.000000\n",
       "max      100.000000"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df[['popularity']].describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Based on the metrics immediately **above**, I'm going to drop all songs with a popularity rating of less than 25 out of 100. I landed on a value of 25 because, even though the bottom 25th percentile is a score of 43 or below, I do want to retain some of the lower values to account for niche music that may not get as many streams or have the level of exposure as popular commercially released music. Given that the point of the project is to help new artists market themselves, I don't believe there's much value in keeping tracks with extremely low popularity scores."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Dropping all songs with a popularity rating of less than 25 out of 100.\n",
    "df = df[(df['popularity'] >= 25)]\n",
    "\n",
    "# Dropping all rows where the \"artists\" feature contains the word \"Cast\" and \"Broadway Company\".\n",
    "# These are Broadway and musical film cast recordings, which are not within the scope of this project's goals. \n",
    "df = df[(df['artists'].str.contains('Cast', case=False)) == False]\n",
    "df = df[(df['artists'].str.contains('Broadway Company', case=False)) == False]\n",
    "\n",
    "# Similarly dropping artists containing the phrase \"Motion Picture\".\n",
    "df = df[(df['artists'].str.contains('Motion Picture', case=False)) == False]\n",
    "\n",
    "# I also noticed some Charlie Chaplin dialogue recordings in the dataframe. Dropping here.\n",
    "df = df[(df['artists'].str.contains('Charlie Chaplin', case=False)) == False]\n",
    "\n",
    "# Same with \"Legally Blonde\"\n",
    "df = df[(df['artists'].str.contains('Legally Blonde', case=False)) == False]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(58409, 21)"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**BELOW:** Because the number of songs included in the dataframe differ so much from artist to artist, looking at popularity this way will not be representative of an artist's overall popularity. However, we can still glean a lot about marketability on the song level, without relying too heavily on an artist's overall popularity."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>count</th>\n",
       "      <th>mean</th>\n",
       "      <th>std</th>\n",
       "      <th>min</th>\n",
       "      <th>25%</th>\n",
       "      <th>50%</th>\n",
       "      <th>75%</th>\n",
       "      <th>max</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>artists</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>[\"Adolescent's Orquesta\"]</th>\n",
       "      <td>8.0</td>\n",
       "      <td>62.375</td>\n",
       "      <td>4.206712</td>\n",
       "      <td>54.0</td>\n",
       "      <td>61.5</td>\n",
       "      <td>62.0</td>\n",
       "      <td>66.0</td>\n",
       "      <td>67.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>[\"Alexander O'Neal\"]</th>\n",
       "      <td>1.0</td>\n",
       "      <td>35.000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>35.0</td>\n",
       "      <td>35.0</td>\n",
       "      <td>35.0</td>\n",
       "      <td>35.0</td>\n",
       "      <td>35.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>[\"Amy Hanaiali'i\", 'Willie K']</th>\n",
       "      <td>1.0</td>\n",
       "      <td>35.000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>35.0</td>\n",
       "      <td>35.0</td>\n",
       "      <td>35.0</td>\n",
       "      <td>35.0</td>\n",
       "      <td>35.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>[\"Auli'i Cravalho\", 'Vai Mahina', \"Olivia Foa'i\", \"Opetaia Foa'i\", 'Matthew Ineleo']</th>\n",
       "      <td>1.0</td>\n",
       "      <td>62.000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>62.0</td>\n",
       "      <td>62.0</td>\n",
       "      <td>62.0</td>\n",
       "      <td>62.0</td>\n",
       "      <td>62.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>[\"Auli'i Cravalho\"]</th>\n",
       "      <td>2.0</td>\n",
       "      <td>68.000</td>\n",
       "      <td>8.485281</td>\n",
       "      <td>62.0</td>\n",
       "      <td>65.0</td>\n",
       "      <td>68.0</td>\n",
       "      <td>71.0</td>\n",
       "      <td>74.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>['那英']</th>\n",
       "      <td>1.0</td>\n",
       "      <td>49.000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>49.0</td>\n",
       "      <td>49.0</td>\n",
       "      <td>49.0</td>\n",
       "      <td>49.0</td>\n",
       "      <td>49.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>['邰正宵']</th>\n",
       "      <td>1.0</td>\n",
       "      <td>41.000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>41.0</td>\n",
       "      <td>41.0</td>\n",
       "      <td>41.0</td>\n",
       "      <td>41.0</td>\n",
       "      <td>41.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>['須田景凪']</th>\n",
       "      <td>1.0</td>\n",
       "      <td>67.000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>67.0</td>\n",
       "      <td>67.0</td>\n",
       "      <td>67.0</td>\n",
       "      <td>67.0</td>\n",
       "      <td>67.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>['黃品源']</th>\n",
       "      <td>1.0</td>\n",
       "      <td>48.000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>48.0</td>\n",
       "      <td>48.0</td>\n",
       "      <td>48.0</td>\n",
       "      <td>48.0</td>\n",
       "      <td>48.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>['黑豹']</th>\n",
       "      <td>1.0</td>\n",
       "      <td>35.000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>35.0</td>\n",
       "      <td>35.0</td>\n",
       "      <td>35.0</td>\n",
       "      <td>35.0</td>\n",
       "      <td>35.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>18445 rows × 8 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                    count    mean       std  \\\n",
       "artists                                                                       \n",
       "[\"Adolescent's Orquesta\"]                             8.0  62.375  4.206712   \n",
       "[\"Alexander O'Neal\"]                                  1.0  35.000       NaN   \n",
       "[\"Amy Hanaiali'i\", 'Willie K']                        1.0  35.000       NaN   \n",
       "[\"Auli'i Cravalho\", 'Vai Mahina', \"Olivia Foa'i...    1.0  62.000       NaN   \n",
       "[\"Auli'i Cravalho\"]                                   2.0  68.000  8.485281   \n",
       "...                                                   ...     ...       ...   \n",
       "['那英']                                                1.0  49.000       NaN   \n",
       "['邰正宵']                                               1.0  41.000       NaN   \n",
       "['須田景凪']                                              1.0  67.000       NaN   \n",
       "['黃品源']                                               1.0  48.000       NaN   \n",
       "['黑豹']                                                1.0  35.000       NaN   \n",
       "\n",
       "                                                     min   25%   50%   75%  \\\n",
       "artists                                                                      \n",
       "[\"Adolescent's Orquesta\"]                           54.0  61.5  62.0  66.0   \n",
       "[\"Alexander O'Neal\"]                                35.0  35.0  35.0  35.0   \n",
       "[\"Amy Hanaiali'i\", 'Willie K']                      35.0  35.0  35.0  35.0   \n",
       "[\"Auli'i Cravalho\", 'Vai Mahina', \"Olivia Foa'i...  62.0  62.0  62.0  62.0   \n",
       "[\"Auli'i Cravalho\"]                                 62.0  65.0  68.0  71.0   \n",
       "...                                                  ...   ...   ...   ...   \n",
       "['那英']                                              49.0  49.0  49.0  49.0   \n",
       "['邰正宵']                                             41.0  41.0  41.0  41.0   \n",
       "['須田景凪']                                            67.0  67.0  67.0  67.0   \n",
       "['黃品源']                                             48.0  48.0  48.0  48.0   \n",
       "['黑豹']                                              35.0  35.0  35.0  35.0   \n",
       "\n",
       "                                                     max  \n",
       "artists                                                   \n",
       "[\"Adolescent's Orquesta\"]                           67.0  \n",
       "[\"Alexander O'Neal\"]                                35.0  \n",
       "[\"Amy Hanaiali'i\", 'Willie K']                      35.0  \n",
       "[\"Auli'i Cravalho\", 'Vai Mahina', \"Olivia Foa'i...  62.0  \n",
       "[\"Auli'i Cravalho\"]                                 74.0  \n",
       "...                                                  ...  \n",
       "['那英']                                              49.0  \n",
       "['邰正宵']                                             41.0  \n",
       "['須田景凪']                                            67.0  \n",
       "['黃品源']                                             48.0  \n",
       "['黑豹']                                              35.0  \n",
       "\n",
       "[18445 rows x 8 columns]"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Looking at popularity statistics based on artist. See caveat above.\n",
    "df['popularity'].groupby(by=df['artists']).describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>artists</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>58409</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>unique</th>\n",
       "      <td>18445</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>top</th>\n",
       "      <td>['Taylor Swift']</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>freq</th>\n",
       "      <td>180</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 artists\n",
       "count              58409\n",
       "unique             18445\n",
       "top     ['Taylor Swift']\n",
       "freq                 180"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# After cleaning, the dataframe contains 18,493 distinct artists, and Taylor Swift is the most frequently occurring artist\n",
    "df[['artists']].describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**BELOW:** Looking at track duration summary statistics, and then filtering out entries for extremely short and extremely long tracks. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Engineering a new column to see track length in seconds, for easier analysis, and rounding to one decimal place.\n",
    "df['duration_s'] = round((df['duration_ms'] / 1000), 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Now that I have the duration in seconds, dropping duration in ms feature\n",
    "df.drop(columns='duration_ms', inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "count    58409.000000\n",
       "mean       235.045264\n",
       "std         87.959997\n",
       "min         30.100000\n",
       "25%        191.400000\n",
       "50%        225.500000\n",
       "75%        265.800000\n",
       "max       4270.000000\n",
       "Name: duration_s, dtype: float64"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['duration_s'].describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Based on most current popular music, I'm going to limit my datafram to track lengths between 75 seconds and 420 seconds, or roughly a minute and a quarter to seven minutes in length."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Filtering out tracks less than 75 seconds in length and greater than 420 seconds in length.\n",
    "df = df[(df['duration_s'] >= 75.0) & (df['duration_s'] <= 420.0)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "count    56798.000000\n",
       "mean       229.189033\n",
       "std         57.664611\n",
       "min         75.000000\n",
       "25%        191.300000\n",
       "50%        224.500000\n",
       "75%        262.500000\n",
       "max        420.000000\n",
       "Name: duration_s, dtype: float64"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['duration_s'].describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(56798, 21)"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**BELOW:** Creating another version of my dataframe with `full_key` dummified for modeling purposes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>A minor</th>\n",
       "      <th>A# major</th>\n",
       "      <th>A# minor</th>\n",
       "      <th>B major</th>\n",
       "      <th>B minor</th>\n",
       "      <th>C major</th>\n",
       "      <th>C minor</th>\n",
       "      <th>C# major</th>\n",
       "      <th>C# minor</th>\n",
       "      <th>D major</th>\n",
       "      <th>...</th>\n",
       "      <th>E major</th>\n",
       "      <th>E minor</th>\n",
       "      <th>F major</th>\n",
       "      <th>F minor</th>\n",
       "      <th>F# major</th>\n",
       "      <th>F# minor</th>\n",
       "      <th>G major</th>\n",
       "      <th>G minor</th>\n",
       "      <th>G# major</th>\n",
       "      <th>G# minor</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>13606</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13607</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13608</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13609</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13610</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 23 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       A minor  A# major  A# minor  B major  B minor  C major  C minor  \\\n",
       "13606        0         0         0        0        0        0        0   \n",
       "13607        0         0         0        0        0        0        0   \n",
       "13608        0         0         0        0        0        0        0   \n",
       "13609        0         0         0        0        0        0        0   \n",
       "13610        0         0         0        0        0        0        0   \n",
       "\n",
       "       C# major  C# minor  D major  ...  E major  E minor  F major  F minor  \\\n",
       "13606         0         0        0  ...        1        0        0        0   \n",
       "13607         0         0        0  ...        0        0        0        0   \n",
       "13608         0         0        0  ...        0        0        0        0   \n",
       "13609         0         0        0  ...        0        0        1        0   \n",
       "13610         0         0        0  ...        0        0        1        0   \n",
       "\n",
       "       F# major  F# minor  G major  G minor  G# major  G# minor  \n",
       "13606         0         0        0        0         0         0  \n",
       "13607         0         0        0        0         0         1  \n",
       "13608         0         0        0        0         1         0  \n",
       "13609         0         0        0        0         0         0  \n",
       "13610         0         0        0        0         0         0  \n",
       "\n",
       "[5 rows x 23 columns]"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "key_dummies = pd.get_dummies(df['full_key'], drop_first=True)\n",
    "key_dummies.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Merging the dummified keys back to the original dataframe\n",
    "dummy_df = pd.concat((df, key_dummies), axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['name', 'artists', 'tempo', 'key', 'mode', 'full_key', 'secondary_key',\n",
       "       'secondary_mode', 'secondary_full_key', 'loudness', 'energy',\n",
       "       'instrumentalness', 'speechiness', 'acousticness', 'danceability',\n",
       "       'valence', 'popularity', 'liveness', 'year', 'id', 'duration_s'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['name', 'artists', 'tempo', 'key', 'mode', 'full_key', 'secondary_key',\n",
       "       'secondary_mode', 'secondary_full_key', 'loudness', 'energy',\n",
       "       'instrumentalness', 'speechiness', 'acousticness', 'danceability',\n",
       "       'valence', 'popularity', 'liveness', 'year', 'id', 'duration_s',\n",
       "       'A minor', 'A# major', 'A# minor', 'B major', 'B minor', 'C major',\n",
       "       'C minor', 'C# major', 'C# minor', 'D major', 'D minor', 'D# major',\n",
       "       'D# minor', 'E major', 'E minor', 'F major', 'F minor', 'F# major',\n",
       "       'F# minor', 'G major', 'G minor', 'G# major', 'G# minor'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dummy_df.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Reordering columns in my dummied df, again for clarity\n",
    "dummy_df = dummy_df[['name', 'artists', 'tempo', 'key', 'mode', 'full_key', 'secondary_key', 'secondary_mode', 'secondary_full_key', 'A minor', \n",
    "                     'A# major', 'A# minor', 'B major', 'B minor', 'C major', 'C minor', 'C# major', 'C# minor', 'D major', 'D minor', 'D# major', \n",
    "                     'D# minor', 'E major', 'E minor', 'F major', 'F minor', 'F# major', 'F# minor', 'G major', 'G minor', 'G# major', 'G# minor', \n",
    "                     'loudness', 'duration_s', 'energy', 'instrumentalness', 'speechiness', 'acousticness', 'danceability', 'valence', 'popularity', \n",
    "                     'liveness', 'year', 'id']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>artists</th>\n",
       "      <th>tempo</th>\n",
       "      <th>key</th>\n",
       "      <th>mode</th>\n",
       "      <th>full_key</th>\n",
       "      <th>secondary_key</th>\n",
       "      <th>secondary_mode</th>\n",
       "      <th>secondary_full_key</th>\n",
       "      <th>A minor</th>\n",
       "      <th>...</th>\n",
       "      <th>energy</th>\n",
       "      <th>instrumentalness</th>\n",
       "      <th>speechiness</th>\n",
       "      <th>acousticness</th>\n",
       "      <th>danceability</th>\n",
       "      <th>valence</th>\n",
       "      <th>popularity</th>\n",
       "      <th>liveness</th>\n",
       "      <th>year</th>\n",
       "      <th>id</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>13606</th>\n",
       "      <td>Thunderstruck</td>\n",
       "      <td>['AC/DC']</td>\n",
       "      <td>133.5</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>E major</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>C# minor</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.890</td>\n",
       "      <td>0.011700</td>\n",
       "      <td>0.0364</td>\n",
       "      <td>0.000147</td>\n",
       "      <td>0.502</td>\n",
       "      <td>0.259</td>\n",
       "      <td>83</td>\n",
       "      <td>0.2170</td>\n",
       "      <td>1990</td>\n",
       "      <td>57bgtoPSgt236HzfBOd8kj</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13607</th>\n",
       "      <td>Carol of the Bells</td>\n",
       "      <td>['Mykola Dmytrovych Leontovych', 'John Williams']</td>\n",
       "      <td>46.7</td>\n",
       "      <td>8</td>\n",
       "      <td>0</td>\n",
       "      <td>G# minor</td>\n",
       "      <td>11</td>\n",
       "      <td>1</td>\n",
       "      <td>B major</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.106</td>\n",
       "      <td>0.029200</td>\n",
       "      <td>0.0448</td>\n",
       "      <td>0.994000</td>\n",
       "      <td>0.418</td>\n",
       "      <td>0.800</td>\n",
       "      <td>73</td>\n",
       "      <td>0.1790</td>\n",
       "      <td>1990</td>\n",
       "      <td>4tHqQMWSqmL6YjXwsqthDI</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13608</th>\n",
       "      <td>Man in the Box</td>\n",
       "      <td>['Alice In Chains']</td>\n",
       "      <td>106.4</td>\n",
       "      <td>8</td>\n",
       "      <td>1</td>\n",
       "      <td>G# major</td>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>F minor</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.791</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0453</td>\n",
       "      <td>0.000450</td>\n",
       "      <td>0.355</td>\n",
       "      <td>0.758</td>\n",
       "      <td>73</td>\n",
       "      <td>0.0969</td>\n",
       "      <td>1990</td>\n",
       "      <td>6gZVQvQZOFpzIy3HblJ20F</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13609</th>\n",
       "      <td>Don't Go Breaking My Heart</td>\n",
       "      <td>['Elton John', 'Kiki Dee']</td>\n",
       "      <td>131.2</td>\n",
       "      <td>5</td>\n",
       "      <td>1</td>\n",
       "      <td>F major</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>D minor</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.863</td>\n",
       "      <td>0.000006</td>\n",
       "      <td>0.0380</td>\n",
       "      <td>0.177000</td>\n",
       "      <td>0.742</td>\n",
       "      <td>0.776</td>\n",
       "      <td>69</td>\n",
       "      <td>0.0699</td>\n",
       "      <td>1990</td>\n",
       "      <td>5pKJtX4wBeby9qIfFhyOJj</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13610</th>\n",
       "      <td>This Christmas</td>\n",
       "      <td>['Donny Hathaway']</td>\n",
       "      <td>93.0</td>\n",
       "      <td>5</td>\n",
       "      <td>1</td>\n",
       "      <td>F major</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>D minor</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.645</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0376</td>\n",
       "      <td>0.292000</td>\n",
       "      <td>0.634</td>\n",
       "      <td>0.677</td>\n",
       "      <td>70</td>\n",
       "      <td>0.2460</td>\n",
       "      <td>1990</td>\n",
       "      <td>38xhBO2AKrJnjdjVnhJES6</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 44 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                             name  \\\n",
       "13606               Thunderstruck   \n",
       "13607          Carol of the Bells   \n",
       "13608              Man in the Box   \n",
       "13609  Don't Go Breaking My Heart   \n",
       "13610              This Christmas   \n",
       "\n",
       "                                                 artists  tempo  key  mode  \\\n",
       "13606                                          ['AC/DC']  133.5    4     1   \n",
       "13607  ['Mykola Dmytrovych Leontovych', 'John Williams']   46.7    8     0   \n",
       "13608                                ['Alice In Chains']  106.4    8     1   \n",
       "13609                         ['Elton John', 'Kiki Dee']  131.2    5     1   \n",
       "13610                                 ['Donny Hathaway']   93.0    5     1   \n",
       "\n",
       "       full_key  secondary_key  secondary_mode secondary_full_key  A minor  \\\n",
       "13606   E major              1               0           C# minor        0   \n",
       "13607  G# minor             11               1            B major        0   \n",
       "13608  G# major              5               0            F minor        0   \n",
       "13609   F major              2               0            D minor        0   \n",
       "13610   F major              2               0            D minor        0   \n",
       "\n",
       "       ...  energy  instrumentalness  speechiness  acousticness  danceability  \\\n",
       "13606  ...   0.890          0.011700       0.0364      0.000147         0.502   \n",
       "13607  ...   0.106          0.029200       0.0448      0.994000         0.418   \n",
       "13608  ...   0.791          0.000000       0.0453      0.000450         0.355   \n",
       "13609  ...   0.863          0.000006       0.0380      0.177000         0.742   \n",
       "13610  ...   0.645          0.000000       0.0376      0.292000         0.634   \n",
       "\n",
       "       valence  popularity  liveness  year                      id  \n",
       "13606    0.259          83    0.2170  1990  57bgtoPSgt236HzfBOd8kj  \n",
       "13607    0.800          73    0.1790  1990  4tHqQMWSqmL6YjXwsqthDI  \n",
       "13608    0.758          73    0.0969  1990  6gZVQvQZOFpzIy3HblJ20F  \n",
       "13609    0.776          69    0.0699  1990  5pKJtX4wBeby9qIfFhyOJj  \n",
       "13610    0.677          70    0.2460  1990  38xhBO2AKrJnjdjVnhJES6  \n",
       "\n",
       "[5 rows x 44 columns]"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dummy_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>artists</th>\n",
       "      <th>tempo</th>\n",
       "      <th>key</th>\n",
       "      <th>mode</th>\n",
       "      <th>full_key</th>\n",
       "      <th>secondary_key</th>\n",
       "      <th>secondary_mode</th>\n",
       "      <th>secondary_full_key</th>\n",
       "      <th>loudness</th>\n",
       "      <th>...</th>\n",
       "      <th>instrumentalness</th>\n",
       "      <th>speechiness</th>\n",
       "      <th>acousticness</th>\n",
       "      <th>danceability</th>\n",
       "      <th>valence</th>\n",
       "      <th>popularity</th>\n",
       "      <th>liveness</th>\n",
       "      <th>year</th>\n",
       "      <th>id</th>\n",
       "      <th>duration_s</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>13606</th>\n",
       "      <td>Thunderstruck</td>\n",
       "      <td>['AC/DC']</td>\n",
       "      <td>133.5</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>E major</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>C# minor</td>\n",
       "      <td>-5.175</td>\n",
       "      <td>...</td>\n",
       "      <td>0.011700</td>\n",
       "      <td>0.0364</td>\n",
       "      <td>0.000147</td>\n",
       "      <td>0.502</td>\n",
       "      <td>0.259</td>\n",
       "      <td>83</td>\n",
       "      <td>0.2170</td>\n",
       "      <td>1990</td>\n",
       "      <td>57bgtoPSgt236HzfBOd8kj</td>\n",
       "      <td>292.9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13607</th>\n",
       "      <td>Carol of the Bells</td>\n",
       "      <td>['Mykola Dmytrovych Leontovych', 'John Williams']</td>\n",
       "      <td>46.7</td>\n",
       "      <td>8</td>\n",
       "      <td>0</td>\n",
       "      <td>G# minor</td>\n",
       "      <td>11</td>\n",
       "      <td>1</td>\n",
       "      <td>B major</td>\n",
       "      <td>-22.507</td>\n",
       "      <td>...</td>\n",
       "      <td>0.029200</td>\n",
       "      <td>0.0448</td>\n",
       "      <td>0.994000</td>\n",
       "      <td>0.418</td>\n",
       "      <td>0.800</td>\n",
       "      <td>73</td>\n",
       "      <td>0.1790</td>\n",
       "      <td>1990</td>\n",
       "      <td>4tHqQMWSqmL6YjXwsqthDI</td>\n",
       "      <td>85.3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13608</th>\n",
       "      <td>Man in the Box</td>\n",
       "      <td>['Alice In Chains']</td>\n",
       "      <td>106.4</td>\n",
       "      <td>8</td>\n",
       "      <td>1</td>\n",
       "      <td>G# major</td>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>F minor</td>\n",
       "      <td>-7.565</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0453</td>\n",
       "      <td>0.000450</td>\n",
       "      <td>0.355</td>\n",
       "      <td>0.758</td>\n",
       "      <td>73</td>\n",
       "      <td>0.0969</td>\n",
       "      <td>1990</td>\n",
       "      <td>6gZVQvQZOFpzIy3HblJ20F</td>\n",
       "      <td>285.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13609</th>\n",
       "      <td>Don't Go Breaking My Heart</td>\n",
       "      <td>['Elton John', 'Kiki Dee']</td>\n",
       "      <td>131.2</td>\n",
       "      <td>5</td>\n",
       "      <td>1</td>\n",
       "      <td>F major</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>D minor</td>\n",
       "      <td>-7.823</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000006</td>\n",
       "      <td>0.0380</td>\n",
       "      <td>0.177000</td>\n",
       "      <td>0.742</td>\n",
       "      <td>0.776</td>\n",
       "      <td>69</td>\n",
       "      <td>0.0699</td>\n",
       "      <td>1990</td>\n",
       "      <td>5pKJtX4wBeby9qIfFhyOJj</td>\n",
       "      <td>271.1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13610</th>\n",
       "      <td>This Christmas</td>\n",
       "      <td>['Donny Hathaway']</td>\n",
       "      <td>93.0</td>\n",
       "      <td>5</td>\n",
       "      <td>1</td>\n",
       "      <td>F major</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>D minor</td>\n",
       "      <td>-9.200</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0376</td>\n",
       "      <td>0.292000</td>\n",
       "      <td>0.634</td>\n",
       "      <td>0.677</td>\n",
       "      <td>70</td>\n",
       "      <td>0.2460</td>\n",
       "      <td>1990</td>\n",
       "      <td>38xhBO2AKrJnjdjVnhJES6</td>\n",
       "      <td>231.2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 21 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                             name  \\\n",
       "13606               Thunderstruck   \n",
       "13607          Carol of the Bells   \n",
       "13608              Man in the Box   \n",
       "13609  Don't Go Breaking My Heart   \n",
       "13610              This Christmas   \n",
       "\n",
       "                                                 artists  tempo  key  mode  \\\n",
       "13606                                          ['AC/DC']  133.5    4     1   \n",
       "13607  ['Mykola Dmytrovych Leontovych', 'John Williams']   46.7    8     0   \n",
       "13608                                ['Alice In Chains']  106.4    8     1   \n",
       "13609                         ['Elton John', 'Kiki Dee']  131.2    5     1   \n",
       "13610                                 ['Donny Hathaway']   93.0    5     1   \n",
       "\n",
       "       full_key  secondary_key  secondary_mode secondary_full_key  loudness  \\\n",
       "13606   E major              1               0           C# minor    -5.175   \n",
       "13607  G# minor             11               1            B major   -22.507   \n",
       "13608  G# major              5               0            F minor    -7.565   \n",
       "13609   F major              2               0            D minor    -7.823   \n",
       "13610   F major              2               0            D minor    -9.200   \n",
       "\n",
       "       ...  instrumentalness  speechiness  acousticness  danceability  \\\n",
       "13606  ...          0.011700       0.0364      0.000147         0.502   \n",
       "13607  ...          0.029200       0.0448      0.994000         0.418   \n",
       "13608  ...          0.000000       0.0453      0.000450         0.355   \n",
       "13609  ...          0.000006       0.0380      0.177000         0.742   \n",
       "13610  ...          0.000000       0.0376      0.292000         0.634   \n",
       "\n",
       "       valence  popularity  liveness  year                      id duration_s  \n",
       "13606    0.259          83    0.2170  1990  57bgtoPSgt236HzfBOd8kj      292.9  \n",
       "13607    0.800          73    0.1790  1990  4tHqQMWSqmL6YjXwsqthDI       85.3  \n",
       "13608    0.758          73    0.0969  1990  6gZVQvQZOFpzIy3HblJ20F      285.2  \n",
       "13609    0.776          69    0.0699  1990  5pKJtX4wBeby9qIfFhyOJj      271.1  \n",
       "13610    0.677          70    0.2460  1990  38xhBO2AKrJnjdjVnhJES6      231.2  \n",
       "\n",
       "[5 rows x 21 columns]"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(56798, 21)"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(56798, 44)"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dummy_df.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "---\n",
    "### Section 3\n",
    "#### Re-Indexing & Exports"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**NOTE:** I began with a dataframe containing 170,653 rows, and have cut it down through cleaning to 56,798 rows. Though this means I'm only keeping about 33% of my original data, I believe this is justified:\n",
    "1. Roughly 60,000 entries should be plenty of data to train a preliminary model.\n",
    "2. The data I removed was done so strategically, and with respect to metrics that I intuitively know carry more weight, given my history in the music industry.\n",
    "3. Limiting the data to recent and at least somewhat popular music ensures that the model will compare your music against relevant songs and artists."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Sorting dataframes by year\n",
    "df.sort_values(by='year', inplace=True)\n",
    "dummy_df.sort_values(by='year', inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Resetting index for final clean dataframes\n",
    "df.reset_index(inplace=True)\n",
    "df.drop(columns='index', inplace=True)\n",
    "\n",
    "dummy_df.reset_index(inplace=True)\n",
    "dummy_df.drop(columns='index', inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>artists</th>\n",
       "      <th>tempo</th>\n",
       "      <th>key</th>\n",
       "      <th>mode</th>\n",
       "      <th>full_key</th>\n",
       "      <th>secondary_key</th>\n",
       "      <th>secondary_mode</th>\n",
       "      <th>secondary_full_key</th>\n",
       "      <th>loudness</th>\n",
       "      <th>...</th>\n",
       "      <th>instrumentalness</th>\n",
       "      <th>speechiness</th>\n",
       "      <th>acousticness</th>\n",
       "      <th>danceability</th>\n",
       "      <th>valence</th>\n",
       "      <th>popularity</th>\n",
       "      <th>liveness</th>\n",
       "      <th>year</th>\n",
       "      <th>id</th>\n",
       "      <th>duration_s</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Thunderstruck</td>\n",
       "      <td>['AC/DC']</td>\n",
       "      <td>133.5</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>E major</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>C# minor</td>\n",
       "      <td>-5.175</td>\n",
       "      <td>...</td>\n",
       "      <td>0.01170</td>\n",
       "      <td>0.0364</td>\n",
       "      <td>0.000147</td>\n",
       "      <td>0.502</td>\n",
       "      <td>0.259</td>\n",
       "      <td>83</td>\n",
       "      <td>0.217</td>\n",
       "      <td>1990</td>\n",
       "      <td>57bgtoPSgt236HzfBOd8kj</td>\n",
       "      <td>292.9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>The Gift of Love</td>\n",
       "      <td>['Bette Midler']</td>\n",
       "      <td>157.5</td>\n",
       "      <td>8</td>\n",
       "      <td>1</td>\n",
       "      <td>G# major</td>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>F minor</td>\n",
       "      <td>-10.765</td>\n",
       "      <td>...</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.0287</td>\n",
       "      <td>0.359000</td>\n",
       "      <td>0.486</td>\n",
       "      <td>0.286</td>\n",
       "      <td>38</td>\n",
       "      <td>0.110</td>\n",
       "      <td>1990</td>\n",
       "      <td>7FUc1xVSKvABmVwI6kS5Y4</td>\n",
       "      <td>241.6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Thelma - Bonus Track</td>\n",
       "      <td>['Paul Simon']</td>\n",
       "      <td>94.0</td>\n",
       "      <td>5</td>\n",
       "      <td>1</td>\n",
       "      <td>F major</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>D minor</td>\n",
       "      <td>-13.367</td>\n",
       "      <td>...</td>\n",
       "      <td>0.08450</td>\n",
       "      <td>0.0770</td>\n",
       "      <td>0.872000</td>\n",
       "      <td>0.710</td>\n",
       "      <td>0.882</td>\n",
       "      <td>29</td>\n",
       "      <td>0.093</td>\n",
       "      <td>1990</td>\n",
       "      <td>7pcEC5r1jVqWGRypo9D7f7</td>\n",
       "      <td>254.7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>How I Need You</td>\n",
       "      <td>['Bad Boys Blue']</td>\n",
       "      <td>123.1</td>\n",
       "      <td>9</td>\n",
       "      <td>0</td>\n",
       "      <td>A minor</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>C major</td>\n",
       "      <td>-12.096</td>\n",
       "      <td>...</td>\n",
       "      <td>0.00347</td>\n",
       "      <td>0.0398</td>\n",
       "      <td>0.072400</td>\n",
       "      <td>0.652</td>\n",
       "      <td>0.963</td>\n",
       "      <td>44</td>\n",
       "      <td>0.119</td>\n",
       "      <td>1990</td>\n",
       "      <td>1yq8h4zD0IDT5X1YTaEwZh</td>\n",
       "      <td>218.3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Nunca Dudes De Mi</td>\n",
       "      <td>['El Golpe']</td>\n",
       "      <td>143.1</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>E major</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>C# minor</td>\n",
       "      <td>-8.606</td>\n",
       "      <td>...</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.0295</td>\n",
       "      <td>0.151000</td>\n",
       "      <td>0.476</td>\n",
       "      <td>0.514</td>\n",
       "      <td>31</td>\n",
       "      <td>0.305</td>\n",
       "      <td>1990</td>\n",
       "      <td>5kNYkLFs3WFFgE6qhfWDEm</td>\n",
       "      <td>176.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 21 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                   name            artists  tempo  key  mode  full_key  \\\n",
       "0         Thunderstruck          ['AC/DC']  133.5    4     1   E major   \n",
       "1      The Gift of Love   ['Bette Midler']  157.5    8     1  G# major   \n",
       "2  Thelma - Bonus Track     ['Paul Simon']   94.0    5     1   F major   \n",
       "3        How I Need You  ['Bad Boys Blue']  123.1    9     0   A minor   \n",
       "4     Nunca Dudes De Mi       ['El Golpe']  143.1    4     1   E major   \n",
       "\n",
       "   secondary_key  secondary_mode secondary_full_key  loudness  ...  \\\n",
       "0              1               0           C# minor    -5.175  ...   \n",
       "1              5               0            F minor   -10.765  ...   \n",
       "2              2               0            D minor   -13.367  ...   \n",
       "3              0               1            C major   -12.096  ...   \n",
       "4              1               0           C# minor    -8.606  ...   \n",
       "\n",
       "   instrumentalness  speechiness  acousticness  danceability  valence  \\\n",
       "0           0.01170       0.0364      0.000147         0.502    0.259   \n",
       "1           0.00000       0.0287      0.359000         0.486    0.286   \n",
       "2           0.08450       0.0770      0.872000         0.710    0.882   \n",
       "3           0.00347       0.0398      0.072400         0.652    0.963   \n",
       "4           0.00000       0.0295      0.151000         0.476    0.514   \n",
       "\n",
       "   popularity  liveness  year                      id duration_s  \n",
       "0          83     0.217  1990  57bgtoPSgt236HzfBOd8kj      292.9  \n",
       "1          38     0.110  1990  7FUc1xVSKvABmVwI6kS5Y4      241.6  \n",
       "2          29     0.093  1990  7pcEC5r1jVqWGRypo9D7f7      254.7  \n",
       "3          44     0.119  1990  1yq8h4zD0IDT5X1YTaEwZh      218.3  \n",
       "4          31     0.305  1990  5kNYkLFs3WFFgE6qhfWDEm      176.0  \n",
       "\n",
       "[5 rows x 21 columns]"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>artists</th>\n",
       "      <th>tempo</th>\n",
       "      <th>key</th>\n",
       "      <th>mode</th>\n",
       "      <th>full_key</th>\n",
       "      <th>secondary_key</th>\n",
       "      <th>secondary_mode</th>\n",
       "      <th>secondary_full_key</th>\n",
       "      <th>A minor</th>\n",
       "      <th>...</th>\n",
       "      <th>energy</th>\n",
       "      <th>instrumentalness</th>\n",
       "      <th>speechiness</th>\n",
       "      <th>acousticness</th>\n",
       "      <th>danceability</th>\n",
       "      <th>valence</th>\n",
       "      <th>popularity</th>\n",
       "      <th>liveness</th>\n",
       "      <th>year</th>\n",
       "      <th>id</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Thunderstruck</td>\n",
       "      <td>['AC/DC']</td>\n",
       "      <td>133.5</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>E major</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>C# minor</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.890</td>\n",
       "      <td>0.01170</td>\n",
       "      <td>0.0364</td>\n",
       "      <td>0.000147</td>\n",
       "      <td>0.502</td>\n",
       "      <td>0.259</td>\n",
       "      <td>83</td>\n",
       "      <td>0.217</td>\n",
       "      <td>1990</td>\n",
       "      <td>57bgtoPSgt236HzfBOd8kj</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>The Gift of Love</td>\n",
       "      <td>['Bette Midler']</td>\n",
       "      <td>157.5</td>\n",
       "      <td>8</td>\n",
       "      <td>1</td>\n",
       "      <td>G# major</td>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>F minor</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.467</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.0287</td>\n",
       "      <td>0.359000</td>\n",
       "      <td>0.486</td>\n",
       "      <td>0.286</td>\n",
       "      <td>38</td>\n",
       "      <td>0.110</td>\n",
       "      <td>1990</td>\n",
       "      <td>7FUc1xVSKvABmVwI6kS5Y4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Thelma - Bonus Track</td>\n",
       "      <td>['Paul Simon']</td>\n",
       "      <td>94.0</td>\n",
       "      <td>5</td>\n",
       "      <td>1</td>\n",
       "      <td>F major</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>D minor</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.529</td>\n",
       "      <td>0.08450</td>\n",
       "      <td>0.0770</td>\n",
       "      <td>0.872000</td>\n",
       "      <td>0.710</td>\n",
       "      <td>0.882</td>\n",
       "      <td>29</td>\n",
       "      <td>0.093</td>\n",
       "      <td>1990</td>\n",
       "      <td>7pcEC5r1jVqWGRypo9D7f7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>How I Need You</td>\n",
       "      <td>['Bad Boys Blue']</td>\n",
       "      <td>123.1</td>\n",
       "      <td>9</td>\n",
       "      <td>0</td>\n",
       "      <td>A minor</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>C major</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>0.670</td>\n",
       "      <td>0.00347</td>\n",
       "      <td>0.0398</td>\n",
       "      <td>0.072400</td>\n",
       "      <td>0.652</td>\n",
       "      <td>0.963</td>\n",
       "      <td>44</td>\n",
       "      <td>0.119</td>\n",
       "      <td>1990</td>\n",
       "      <td>1yq8h4zD0IDT5X1YTaEwZh</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Nunca Dudes De Mi</td>\n",
       "      <td>['El Golpe']</td>\n",
       "      <td>143.1</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>E major</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>C# minor</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.490</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.0295</td>\n",
       "      <td>0.151000</td>\n",
       "      <td>0.476</td>\n",
       "      <td>0.514</td>\n",
       "      <td>31</td>\n",
       "      <td>0.305</td>\n",
       "      <td>1990</td>\n",
       "      <td>5kNYkLFs3WFFgE6qhfWDEm</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 44 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                   name            artists  tempo  key  mode  full_key  \\\n",
       "0         Thunderstruck          ['AC/DC']  133.5    4     1   E major   \n",
       "1      The Gift of Love   ['Bette Midler']  157.5    8     1  G# major   \n",
       "2  Thelma - Bonus Track     ['Paul Simon']   94.0    5     1   F major   \n",
       "3        How I Need You  ['Bad Boys Blue']  123.1    9     0   A minor   \n",
       "4     Nunca Dudes De Mi       ['El Golpe']  143.1    4     1   E major   \n",
       "\n",
       "   secondary_key  secondary_mode secondary_full_key  A minor  ...  energy  \\\n",
       "0              1               0           C# minor        0  ...   0.890   \n",
       "1              5               0            F minor        0  ...   0.467   \n",
       "2              2               0            D minor        0  ...   0.529   \n",
       "3              0               1            C major        1  ...   0.670   \n",
       "4              1               0           C# minor        0  ...   0.490   \n",
       "\n",
       "   instrumentalness  speechiness  acousticness  danceability  valence  \\\n",
       "0           0.01170       0.0364      0.000147         0.502    0.259   \n",
       "1           0.00000       0.0287      0.359000         0.486    0.286   \n",
       "2           0.08450       0.0770      0.872000         0.710    0.882   \n",
       "3           0.00347       0.0398      0.072400         0.652    0.963   \n",
       "4           0.00000       0.0295      0.151000         0.476    0.514   \n",
       "\n",
       "   popularity  liveness  year                      id  \n",
       "0          83     0.217  1990  57bgtoPSgt236HzfBOd8kj  \n",
       "1          38     0.110  1990  7FUc1xVSKvABmVwI6kS5Y4  \n",
       "2          29     0.093  1990  7pcEC5r1jVqWGRypo9D7f7  \n",
       "3          44     0.119  1990  1yq8h4zD0IDT5X1YTaEwZh  \n",
       "4          31     0.305  1990  5kNYkLFs3WFFgE6qhfWDEm  \n",
       "\n",
       "[5 rows x 44 columns]"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dummy_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(56798, 21)"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(56798, 44)"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dummy_df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.isnull().sum().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dummy_df.isnull().sum().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2018    0.034526\n",
       "2002    0.033558\n",
       "2001    0.033293\n",
       "1998    0.033205\n",
       "1996    0.033100\n",
       "2006    0.032994\n",
       "2010    0.032977\n",
       "1990    0.032906\n",
       "2017    0.032888\n",
       "2014    0.032888\n",
       "2011    0.032871\n",
       "1995    0.032677\n",
       "1997    0.032642\n",
       "1994    0.032572\n",
       "2007    0.032396\n",
       "2004    0.032360\n",
       "2000    0.032237\n",
       "1992    0.032219\n",
       "2005    0.032202\n",
       "1993    0.032184\n",
       "2009    0.032167\n",
       "2019    0.032043\n",
       "2008    0.031709\n",
       "2003    0.031691\n",
       "2013    0.031550\n",
       "1999    0.031515\n",
       "2020    0.031462\n",
       "1991    0.031339\n",
       "2015    0.030582\n",
       "2016    0.029649\n",
       "2012    0.029596\n",
       "Name: year, dtype: float64"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Data by year is quite evenly distributed\n",
    "df['year'].value_counts(normalize=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Exporting cleaned dataframes to csv\n",
    "df.to_csv('../data_clean/spotify_kg_master.csv', index=False)\n",
    "dummy_df.to_csv('../data_clean/spotify_kg_dummied.csv', index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}