Pre-processing.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Group Members:\n",
    "- Muhammad Awais `BCS221093`\n",
    "- Syed Umer Sarfaraz `BCS221083`\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Network Intrusion Detection System"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. System Description\n",
    "Network security is an important issue for businesses and organizations in this global village as cyber threats become more advanced and harder to detect. A Network Intrusion Detection System (NIDS) is a tool designed to **monitor** and **analyze** network traffic, looking for **unauthorized access**, **suspicious activities**, or **harmful behavior**. It identified threats in real-time by generating an alert which helps security team take measutes within time. NIDS helps *secure sensitive data, prevent system breaches, and ensure the network stays reliable and protected*.\n",
    "\n",
    "Nowadays almost every business and organizations has its own network. Hence securing that network is a crucial need for them. NIDS system uses modern techniques to distinguish between normal and potentially harmful traffic. They examine network activity, by continously monitoring network traffic both ingoing and outgoing, and by creating logs in real-time. *NIDS plays a key role in protecting against various types of cyber attacks, such as Denial-of-Service (DoS), Land Attacks, and unauthorized logins*. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2. Problem Description\n",
    "Detecting network intrusions in real-time is a challenging task due to the heavy network traffic. Rule-based systems often struggle to handle the dynamic nature of cyber threats, *leading to **increased number of false positives** or **missed detections***. The main problem is to create such mechanism that can effectively classify network traffic into **normal** or **suspicious** activities, thereby minimizing threats without affecting legitimate network activity."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3. How ML Can Help Address the Problem?\n",
    "Machine Learning offers a powerful way to improve intrusion detection compared to *traditional rule-based systems*. Unlike rule-based approaches, which rely on pre-defined rules and signatures, ML can automatically learn patterns from network data. This means *ML systems can detect new and unknown threats that may not follow established patterns*. *They adapt over time, improving their ability to spot suspicious activity as new types of attacks emerge*.\n",
    "\n",
    "Another advantage of ML is its ability to reduce false alarms. Traditional systems often flag normal activities as threats, leading to unnecessary alerts. If ML models are trained accurately, they can reduce false alarms, helping business. By using ML, intrusion detection systems become more **intelligent**, **scalable**, and *capable of keeping up with modern cybersecurity challenges*."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 4. Dataset Description\n",
    "Dataset consist of 33 descriptive features that provides various information related to network activity. The feaures includes different information like content information, tcp connection information, traffic and host related information computed using a time window.\n",
    "\n",
    "> Table below shows brief description of main features:\n",
    "\n",
    "### Dataset Features\n",
    "\n",
    "| **Feature Name**         | **Description**                                                               | **Type**       |\n",
    "|---------------------------|-------------------------------------------------------------------------------|----------------|\n",
    "| 1. `duration`               | How long connection was maintained (in seconds).                                       | numeric     |\n",
    "| 2. `protocol_type`          | Type of protocol used, e.g., `tcp`, `udp`, `icmp`.                           | categorical       |\n",
    "| 3. `service`                | Network service running on Application Layer on the destination side, e.g., `http`, `ftp`, etc.                | categorical       |\n",
    "| 4. `src_bytes`              | Number of data bytes sent from the source to the destination.                | numeric     |\n",
    "| 5. `dst_bytes`              | Number of data bytes sent from the destination to the source.                | numeric     |\n",
    "| 6. `flag`                   | Connection status (normal or error), e.g., `SF`, `REJ`, etc.                 | categorical       |\n",
    "| 7. `land`                   | Land Attack **If source and destination IP addresses are same** (`0` or `1`).  | binary       |\n",
    "| 8. `wrong_fragment`         | Number of malformed or wrong sequenced fragments.                                               | numeric     |\n",
    "| 9. `urgent`                 | Number of urgent packets. (too many might indicate attack)                                                    | numeric     |\n",
    "\n",
    "### Content Features\n",
    "\n",
    "| **Feature Name**         | **Description**                                                               | **Type**       |\n",
    "|---------------------------|-------------------------------------------------------------------------------|----------------|\n",
    "| 10. `hot`                    | Number of \"hot\" indicators in the connection. (e.g., creating files, accessing system directories). High number might indicate attack.                               | numeric     |\n",
    "| 11. `num_failed_logins`      | Number of failed login attempts.                                             | numeric     |\n",
    "| 12. `logged_in`              | Indicates successful login (`1` for yes, `0` for no).                        | binary       |\n",
    "| 13. `num_compromised`        | Number of compromised conditions. System compromised by attacker, like malware infected etc.                                            | numeric     |\n",
    "| 14. `root_shell`             | Indicates if a root shell was obtained (`1` for yes, `0` for no). root access gives admin rights.            | binary       |\n",
    "| 15. `su_attempted`           | Indicates if an \"su root\" command was attempted (`1` for yes, `0` for no). Switching indicates possible exploitation.  | binary       |\n",
    "| 16. `num_file_creations`     | Number of file created.                                          | numeric     |\n",
    "| 17. `num_shells`             | Number of shell prompts initiated. Might indicate scripts running from outside.                                           | numeric     |\n",
    "| 18. `num_access_files`       | Number of operations on access control files.                                | numeric     |\n",
    "| 19. `num_outbound_cmds`      | Number of outbound commands in an FTP session.  Can be an indicator of data exfiltration.                             | numeric     |\n",
    "| 20. `is_host_login`           | Indicates if the loggedin user is a \"host\" (`1` for yes, `0` for no).    | binary       |\n",
    "| 21. `is_guest_login`         | Indicates if the loggedin user is a \"guest\" login (`1` for yes, `0` for no).         | binary       |\n",
    "\n",
    "\n",
    "### Traffic and Host Features\n",
    "\n",
    "| **Feature Name**               | **Description**                                                                                   | **Type**       |\n",
    "|---------------------------------|---------------------------------------------------------------------------------------------------|----------------|\n",
    "| 22. `count`                    | Total number of connections made to the same destination host during a given time window. This feature captures the intensity of interactions with a specific host. | Numeric        |\n",
    "| 23. `srv_count`                | Number of connections to the same service (e.g., HTTP, FTP) over a given time window. | Numeric        |\n",
    "| 24. `serror_rate`              | Percentage of connections with \"SYN\" (synchronize flag) errors, indicating failed connection attempts to the same destination. Helps identify potential SYN flood attacks or connection issues. | Numeric        |\n",
    "| 25. `rerror_rate`              | Percentage of connections with \"REJ\" (reject flag) errors, which can suggest unauthorized or blocked access attempts. | Numeric        |\n",
    "| 26. `same_srv_rate`            | Proportion of connections made to the same service (e.g., HTTP) relative to the total connections for the current host. This provides insight into service consistency. | Numeric        |\n",
    "| 27. `diff_srv_rate`            | Proportion of connections made to different services compared to the total connections for the current host. A high value could suggest diverse or abnormal service access patterns. | Numeric        |\n",
    "| 28. `srv_diff_host_rate`       | Percentage of connections made to different destination hosts while using the same service. A high rate may indicate distributed service requests. | Numeric        |\n",
    "| 29. `dst_host_count`           | Total number of connections directed to a specific destination host during a given time frame. Helps measure traffic volume to the host. | Numeric        |\n",
    "| 30. `dst_host_srv_count`       | Total number of connections to a specific service (e.g., HTTP) on a destination host. Useful for detecting service-focused activities like attacks targeting a specific service. | Numeric        |\n",
    "| 31. `dst_host_diff_srv_rate`   | Proportion of connections made to different services on the destination host. Indicates variability in service requests directed to a host. | Numeric        |\n",
    "| 32. `dst_host_same_src_port_rate` | Proportion of connections using the same source port directed to a specific destination host. High values might suggest persistent activity from a specific port. | Numeric        |\n",
    "| 33. `dst_host_srv_diff_host_rate` | Percentage of connections made to different destination hosts while using the same service on a specific host. Helps identify distributed service requests or load balancing. | Numeric        |\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5. Dataset Collection\n",
    "\n",
    "Is is a publicly available dataset that was used in KDDCup99. it is widely used bu beginners to learn ML techniques to various classifiers on it. KDDCup is Knowledge Discovery and Dataminig competition that is widely known all over the world in the field of DataScience and Machine Learning.\n",
    "\n",
    "The KDDCup dataset is a large dataset with 42 features. It also has refined version for beginners to learn machine learning known as **NSL-KDD** dataset with 33 features. Some features are removed after applying correlation technqiues.\n",
    "The **NSL-KDD** dataset is a refined version of the original KDD Cup 1999 dataset, addressing some of its shortcomings.\n",
    "\n",
    "The dataset is sourced from Kaggle: [NSL-KDD Dataset](https://www.kaggle.com/datasets/kaggleprollc/nsl-kdd99-dataset?resource=download)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 6. Data Preprocessing\n",
    "\n",
    "> Let's perform follwing pre-processing steps:\n",
    "- Convert data to Pandas Dataframe\n",
    "- Check columns\n",
    "- Check data length (shape)\n",
    "- Check for null values.\n",
    "- Check for noisy data.\n",
    "- Check for duplicate records and remove them\n",
    "- Check for outliers\n",
    "- Remove or replace outliers, if necessary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Convert dataset\n",
    "\n",
    "> This code was used to convert dataset to csv format\n",
    "\n",
    "```py\n",
    "import pandas s pd\n",
    "from scipy.io import arff\n",
    "# Convert train dataset to csv\n",
    "data = arff.loadarff('./data/KDDTrain+.arff') \n",
    "df = pd.DataFrame(data[0])\n",
    "\n",
    "df.to_csv('kdd.csv', index=False)\n",
    "print(\"Data saved to kdd.csv\")\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Libs import"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Load dataset and handle exceptions(file not found error)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>duration</th>\n",
       "      <th>protocol_type</th>\n",
       "      <th>service</th>\n",
       "      <th>flag</th>\n",
       "      <th>src_bytes</th>\n",
       "      <th>dst_bytes</th>\n",
       "      <th>land</th>\n",
       "      <th>wrong_fragment</th>\n",
       "      <th>urgent</th>\n",
       "      <th>hot</th>\n",
       "      <th>...</th>\n",
       "      <th>rerror_rate</th>\n",
       "      <th>same_srv_rate</th>\n",
       "      <th>diff_srv_rate</th>\n",
       "      <th>srv_diff_host_rate</th>\n",
       "      <th>dst_host_count</th>\n",
       "      <th>dst_host_srv_count</th>\n",
       "      <th>dst_host_diff_srv_rate</th>\n",
       "      <th>dst_host_same_src_port_rate</th>\n",
       "      <th>dst_host_srv_diff_host_rate</th>\n",
       "      <th>class</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.0</td>\n",
       "      <td>b'tcp'</td>\n",
       "      <td>b'ftp_data'</td>\n",
       "      <td>b'SF'</td>\n",
       "      <td>491.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>b'0'</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>150.0</td>\n",
       "      <td>25.0</td>\n",
       "      <td>0.03</td>\n",
       "      <td>0.17</td>\n",
       "      <td>0.00</td>\n",
       "      <td>b'normal'</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.0</td>\n",
       "      <td>b'udp'</td>\n",
       "      <td>b'other'</td>\n",
       "      <td>b'SF'</td>\n",
       "      <td>146.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>b'0'</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.08</td>\n",
       "      <td>0.15</td>\n",
       "      <td>0.00</td>\n",
       "      <td>255.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.60</td>\n",
       "      <td>0.88</td>\n",
       "      <td>0.00</td>\n",
       "      <td>b'normal'</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.0</td>\n",
       "      <td>b'tcp'</td>\n",
       "      <td>b'private'</td>\n",
       "      <td>b'S0'</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>b'0'</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.05</td>\n",
       "      <td>0.07</td>\n",
       "      <td>0.00</td>\n",
       "      <td>255.0</td>\n",
       "      <td>26.0</td>\n",
       "      <td>0.05</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>b'anomaly'</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.0</td>\n",
       "      <td>b'tcp'</td>\n",
       "      <td>b'http'</td>\n",
       "      <td>b'SF'</td>\n",
       "      <td>232.0</td>\n",
       "      <td>8153.0</td>\n",
       "      <td>b'0'</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>30.0</td>\n",
       "      <td>255.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.03</td>\n",
       "      <td>0.04</td>\n",
       "      <td>b'normal'</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.0</td>\n",
       "      <td>b'tcp'</td>\n",
       "      <td>b'http'</td>\n",
       "      <td>b'SF'</td>\n",
       "      <td>199.0</td>\n",
       "      <td>420.0</td>\n",
       "      <td>b'0'</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.09</td>\n",
       "      <td>255.0</td>\n",
       "      <td>255.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>b'normal'</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 34 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   duration protocol_type      service   flag  src_bytes  dst_bytes  land  \\\n",
       "0       0.0        b'tcp'  b'ftp_data'  b'SF'      491.0        0.0  b'0'   \n",
       "1       0.0        b'udp'     b'other'  b'SF'      146.0        0.0  b'0'   \n",
       "2       0.0        b'tcp'   b'private'  b'S0'        0.0        0.0  b'0'   \n",
       "3       0.0        b'tcp'      b'http'  b'SF'      232.0     8153.0  b'0'   \n",
       "4       0.0        b'tcp'      b'http'  b'SF'      199.0      420.0  b'0'   \n",
       "\n",
       "   wrong_fragment  urgent  hot  ...  rerror_rate same_srv_rate  diff_srv_rate  \\\n",
       "0             0.0     0.0  0.0  ...          0.0          1.00           0.00   \n",
       "1             0.0     0.0  0.0  ...          0.0          0.08           0.15   \n",
       "2             0.0     0.0  0.0  ...          0.0          0.05           0.07   \n",
       "3             0.0     0.0  0.0  ...          0.0          1.00           0.00   \n",
       "4             0.0     0.0  0.0  ...          0.0          1.00           0.00   \n",
       "\n",
       "   srv_diff_host_rate  dst_host_count  dst_host_srv_count  \\\n",
       "0                0.00           150.0                25.0   \n",
       "1                0.00           255.0                 1.0   \n",
       "2                0.00           255.0                26.0   \n",
       "3                0.00            30.0               255.0   \n",
       "4                0.09           255.0               255.0   \n",
       "\n",
       "   dst_host_diff_srv_rate  dst_host_same_src_port_rate  \\\n",
       "0                    0.03                         0.17   \n",
       "1                    0.60                         0.88   \n",
       "2                    0.05                         0.00   \n",
       "3                    0.00                         0.03   \n",
       "4                    0.00                         0.00   \n",
       "\n",
       "   dst_host_srv_diff_host_rate       class  \n",
       "0                         0.00   b'normal'  \n",
       "1                         0.00   b'normal'  \n",
       "2                         0.00  b'anomaly'  \n",
       "3                         0.04   b'normal'  \n",
       "4                         0.00   b'normal'  \n",
       "\n",
       "[5 rows x 34 columns]"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "try:\n",
    "    # LOAD DATASET\n",
    "    df = pd.read_csv('./kdd.csv')\n",
    "    print(df.head(5))  \n",
    "except FileNotFoundError:\n",
    "    print(\"Error: The file 'kdd.csv' was not found.\")\n",
    "except Exception as e:\n",
    "    print(f\"An unexpected error occurred: {e}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The values of some features above are encoded in byte strings instead of plain strings. Let's convert them to plain strings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>duration</th>\n",
       "      <th>protocol_type</th>\n",
       "      <th>service</th>\n",
       "      <th>flag</th>\n",
       "      <th>src_bytes</th>\n",
       "      <th>dst_bytes</th>\n",
       "      <th>land</th>\n",
       "      <th>wrong_fragment</th>\n",
       "      <th>urgent</th>\n",
       "      <th>hot</th>\n",
       "      <th>...</th>\n",
       "      <th>rerror_rate</th>\n",
       "      <th>same_srv_rate</th>\n",
       "      <th>diff_srv_rate</th>\n",
       "      <th>srv_diff_host_rate</th>\n",
       "      <th>dst_host_count</th>\n",
       "      <th>dst_host_srv_count</th>\n",
       "      <th>dst_host_diff_srv_rate</th>\n",
       "      <th>dst_host_same_src_port_rate</th>\n",
       "      <th>dst_host_srv_diff_host_rate</th>\n",
       "      <th>class</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.0</td>\n",
       "      <td>tcp</td>\n",
       "      <td>ftp_data</td>\n",
       "      <td>SF</td>\n",
       "      <td>491.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>150.0</td>\n",
       "      <td>25.0</td>\n",
       "      <td>0.03</td>\n",
       "      <td>0.17</td>\n",
       "      <td>0.00</td>\n",
       "      <td>normal</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.0</td>\n",
       "      <td>udp</td>\n",
       "      <td>other</td>\n",
       "      <td>SF</td>\n",
       "      <td>146.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.08</td>\n",
       "      <td>0.15</td>\n",
       "      <td>0.00</td>\n",
       "      <td>255.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.60</td>\n",
       "      <td>0.88</td>\n",
       "      <td>0.00</td>\n",
       "      <td>normal</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.0</td>\n",
       "      <td>tcp</td>\n",
       "      <td>private</td>\n",
       "      <td>S0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.05</td>\n",
       "      <td>0.07</td>\n",
       "      <td>0.00</td>\n",
       "      <td>255.0</td>\n",
       "      <td>26.0</td>\n",
       "      <td>0.05</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>anomaly</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.0</td>\n",
       "      <td>tcp</td>\n",
       "      <td>http</td>\n",
       "      <td>SF</td>\n",
       "      <td>232.0</td>\n",
       "      <td>8153.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>30.0</td>\n",
       "      <td>255.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.03</td>\n",
       "      <td>0.04</td>\n",
       "      <td>normal</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.0</td>\n",
       "      <td>tcp</td>\n",
       "      <td>http</td>\n",
       "      <td>SF</td>\n",
       "      <td>199.0</td>\n",
       "      <td>420.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.09</td>\n",
       "      <td>255.0</td>\n",
       "      <td>255.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>normal</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 34 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   duration protocol_type   service flag  src_bytes  dst_bytes land  \\\n",
       "0       0.0           tcp  ftp_data   SF      491.0        0.0    0   \n",
       "1       0.0           udp     other   SF      146.0        0.0    0   \n",
       "2       0.0           tcp   private   S0        0.0        0.0    0   \n",
       "3       0.0           tcp      http   SF      232.0     8153.0    0   \n",
       "4       0.0           tcp      http   SF      199.0      420.0    0   \n",
       "\n",
       "   wrong_fragment  urgent  hot  ...  rerror_rate same_srv_rate  diff_srv_rate  \\\n",
       "0             0.0     0.0  0.0  ...          0.0          1.00           0.00   \n",
       "1             0.0     0.0  0.0  ...          0.0          0.08           0.15   \n",
       "2             0.0     0.0  0.0  ...          0.0          0.05           0.07   \n",
       "3             0.0     0.0  0.0  ...          0.0          1.00           0.00   \n",
       "4             0.0     0.0  0.0  ...          0.0          1.00           0.00   \n",
       "\n",
       "   srv_diff_host_rate  dst_host_count  dst_host_srv_count  \\\n",
       "0                0.00           150.0                25.0   \n",
       "1                0.00           255.0                 1.0   \n",
       "2                0.00           255.0                26.0   \n",
       "3                0.00            30.0               255.0   \n",
       "4                0.09           255.0               255.0   \n",
       "\n",
       "   dst_host_diff_srv_rate  dst_host_same_src_port_rate  \\\n",
       "0                    0.03                         0.17   \n",
       "1                    0.60                         0.88   \n",
       "2                    0.05                         0.00   \n",
       "3                    0.00                         0.03   \n",
       "4                    0.00                         0.00   \n",
       "\n",
       "   dst_host_srv_diff_host_rate    class  \n",
       "0                         0.00   normal  \n",
       "1                         0.00   normal  \n",
       "2                         0.00  anomaly  \n",
       "3                         0.04   normal  \n",
       "4                         0.00   normal  \n",
       "\n",
       "[5 rows x 34 columns]"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# byte strings are like this b'value'. To convert them, we can remove \"b'\" from start and \"'\" from end.\n",
    "def clean_byte_string(value):\n",
    "    if isinstance(value, str) and value.startswith(\"b'\"):\n",
    "        return value[2:-1]  # Remove the \"b'\" at the start and the \"'\" at the end\n",
    "    return value\n",
    "\n",
    "for col in df.columns:\n",
    "    df[col] = df[col].apply(clean_byte_string)\n",
    "    \n",
    "    \n",
    "df.head(5)    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Check Columns\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Index(['duration', 'protocol_type', 'service', 'flag', 'src_bytes',\n",
      "       'dst_bytes', 'land', 'wrong_fragment', 'urgent', 'hot',\n",
      "       'num_failed_logins', 'logged_in', 'num_compromised', 'root_shell',\n",
      "       'su_attempted', 'num_file_creations', 'num_shells', 'num_access_files',\n",
      "       'num_outbound_cmds', 'is_host_login', 'is_guest_login', 'count',\n",
      "       'srv_count', 'serror_rate', 'rerror_rate', 'same_srv_rate',\n",
      "       'diff_srv_rate', 'srv_diff_host_rate', 'dst_host_count',\n",
      "       'dst_host_srv_count', 'dst_host_diff_srv_rate',\n",
      "       'dst_host_same_src_port_rate', 'dst_host_srv_diff_host_rate', 'class'],\n",
      "      dtype='object')\n"
     ]
    }
   ],
   "source": [
    "cols = df.columns.to_list()\n",
    "print(df.columns)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Check Shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(124527, 34)"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Count null values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "duration                       0\n",
       "protocol_type                  0\n",
       "service                        0\n",
       "flag                           0\n",
       "src_bytes                      0\n",
       "dst_bytes                      0\n",
       "land                           0\n",
       "wrong_fragment                 0\n",
       "urgent                         0\n",
       "hot                            0\n",
       "num_failed_logins              0\n",
       "logged_in                      0\n",
       "num_compromised                0\n",
       "root_shell                     0\n",
       "su_attempted                   0\n",
       "num_file_creations             0\n",
       "num_shells                     0\n",
       "num_access_files               0\n",
       "num_outbound_cmds              0\n",
       "is_host_login                  0\n",
       "is_guest_login                 0\n",
       "count                          0\n",
       "srv_count                      0\n",
       "serror_rate                    0\n",
       "rerror_rate                    0\n",
       "same_srv_rate                  0\n",
       "diff_srv_rate                  0\n",
       "srv_diff_host_rate             0\n",
       "dst_host_count                 0\n",
       "dst_host_srv_count             0\n",
       "dst_host_diff_srv_rate         0\n",
       "dst_host_same_src_port_rate    0\n",
       "dst_host_srv_diff_host_rate    0\n",
       "class                          0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.isnull().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Check for duplicate records"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of duplicate rows: 1446\n"
     ]
    }
   ],
   "source": [
    "duplicates = df.duplicated()\n",
    "print(f\"Number of duplicate rows: {duplicates.sum()}\")\n",
    "\n",
    "df = df.drop_duplicates()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Separate columns for different types, to apply feature scaling and feature encoding later, if any required"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Numerical Columns: ['duration', 'src_bytes', 'dst_bytes', 'wrong_fragment', 'urgent', 'hot', 'num_failed_logins', 'num_compromised', 'root_shell', 'su_attempted', 'num_file_creations', 'num_shells', 'num_access_files', 'num_outbound_cmds', 'count', 'srv_count', 'serror_rate', 'rerror_rate', 'same_srv_rate', 'diff_srv_rate', 'srv_diff_host_rate', 'dst_host_count', 'dst_host_srv_count', 'dst_host_diff_srv_rate', 'dst_host_same_src_port_rate', 'dst_host_srv_diff_host_rate']\n",
      "Categorical Columns: ['protocol_type', 'service', 'flag', 'land', 'logged_in', 'is_host_login', 'is_guest_login', 'class']\n"
     ]
    }
   ],
   "source": [
    "numerical_cols = df.select_dtypes(include=['int64', 'float64']).columns.to_list()\n",
    "print(\"Numerical Columns:\", numerical_cols)\n",
    "\n",
    "# Select both object and category types for categorical data\n",
    "categorical_cols = df.select_dtypes(include=['object', 'category']).columns.to_list()\n",
    "print(\"Categorical Columns:\", categorical_cols)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Check for class imbalance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "class\n",
      "normal     0.539128\n",
      "anomaly    0.460872\n",
      "Name: proportion, dtype: float64\n"
     ]
    }
   ],
   "source": [
    "# Check class distribution\n",
    "print(df['class'].value_counts(normalize=True))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Checking for number of outliers in numerical features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Outliers in duration: 9998\n",
      "Outliers in src_bytes: 13488\n",
      "Outliers in dst_bytes: 23209\n",
      "Outliers in wrong_fragment: 1072\n",
      "Outliers in urgent: 9\n",
      "Outliers in hot: 2475\n",
      "Outliers in num_failed_logins: 122\n",
      "Outliers in num_compromised: 1094\n",
      "Outliers in root_shell: 169\n",
      "Outliers in su_attempted: 80\n",
      "Outliers in num_file_creations: 287\n",
      "Outliers in num_shells: 47\n",
      "Outliers in num_access_files: 371\n",
      "Outliers in num_outbound_cmds: 0\n",
      "Outliers in count: 2534\n",
      "Outliers in srv_count: 11898\n",
      "Outliers in serror_rate: 0\n",
      "Outliers in rerror_rate: 15154\n",
      "Outliers in same_srv_rate: 0\n",
      "Outliers in diff_srv_rate: 7272\n",
      "Outliers in srv_diff_host_rate: 28295\n",
      "Outliers in dst_host_count: 0\n",
      "Outliers in dst_host_srv_count: 0\n",
      "Outliers in dst_host_diff_srv_rate: 9691\n",
      "Outliers in dst_host_same_src_port_rate: 24666\n",
      "Outliers in dst_host_srv_diff_host_rate: 11334\n"
     ]
    }
   ],
   "source": [
    "for col in numerical_cols:\n",
    "    Q1 = df[col].quantile(0.25)\n",
    "    Q3 = df[col].quantile(0.75)\n",
    "    IQR = Q3 - Q1\n",
    "    lower_bound = Q1 - 1.5 * IQR\n",
    "    upper_bound = Q3 + 1.5 * IQR\n",
    "\n",
    "    # Flag outliers\n",
    "    outliers = ((df[col] < lower_bound) | (df[col] > upper_bound)).sum()\n",
    "    print(f\"Outliers in {col}: {outliers}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Outliers removal\n",
    "\n",
    "Remove records with outlier values for some features, where removing them does not affect data.\n",
    "While other outliers will be replaced with median values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(111856, 34)"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# REMOVING OUTLIERS\n",
    "columns_to_remove_outliers = [\n",
    "    'duration', \n",
    "    'wrong_fragment', \n",
    "    'urgent', \n",
    "    'hot', \n",
    "    'num_failed_logins', \n",
    "    'num_compromised', \n",
    "    'root_shell', \n",
    "    'su_attempted', \n",
    "    'num_file_creations', \n",
    "    'num_shells', \n",
    "    'num_access_files'\n",
    "]\n",
    "\n",
    "for col in columns_to_remove_outliers:\n",
    "    Q1 = df[col].quantile(0.25)\n",
    "    Q3 = df[col].quantile(0.75)\n",
    "    IQR = Q3 - Q1\n",
    "    lower_bound = Q1 - 1.5 * IQR\n",
    "    upper_bound = Q3 + 1.5 * IQR\n",
    "    df = df[(df[col] >= lower_bound) & (df[col] <= upper_bound)]\n",
    "    \n",
    "df.shape    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Replacing outliers with median values, where removing records is not good approach. For example, some features are counts, like number of connections, so removing them will change our dataset originality."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Outliers replaced with median values.\n"
     ]
    }
   ],
   "source": [
    "# REPLACING OUTLIERS WITH MEDIAN VALUES\n",
    "columns_to_replace_outliers = [\n",
    "    'src_bytes',\n",
    "    'dst_bytes',\n",
    "    'count',\n",
    "    'srv_count',\n",
    "    'rerror_rate',\n",
    "    'diff_srv_rate',\n",
    "    'srv_diff_host_rate',\n",
    "    'dst_host_diff_srv_rate',\n",
    "    'dst_host_same_src_port_rate',\n",
    "    'dst_host_srv_diff_host_rate'\n",
    "]\n",
    "\n",
    "for col in columns_to_replace_outliers:\n",
    "    Q1 = df[col].quantile(0.25)\n",
    "    Q3 = df[col].quantile(0.75)\n",
    "    IQR = Q3 - Q1\n",
    "    lower_bound = Q1 - 1.5 * IQR\n",
    "    upper_bound = Q3 + 1.5 * IQR\n",
    "\n",
    "    median_value = df[col].median()\n",
    "    df[col] = df[col].apply(lambda x: median_value if x < lower_bound or x > upper_bound else x)\n",
    "\n",
    "print(\"Outliers replaced with median values.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Shape after removal of some rows"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SHAPE  = (111856, 34)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>duration</th>\n",
       "      <th>protocol_type</th>\n",
       "      <th>service</th>\n",
       "      <th>flag</th>\n",
       "      <th>src_bytes</th>\n",
       "      <th>dst_bytes</th>\n",
       "      <th>land</th>\n",
       "      <th>wrong_fragment</th>\n",
       "      <th>urgent</th>\n",
       "      <th>hot</th>\n",
       "      <th>...</th>\n",
       "      <th>rerror_rate</th>\n",
       "      <th>same_srv_rate</th>\n",
       "      <th>diff_srv_rate</th>\n",
       "      <th>srv_diff_host_rate</th>\n",
       "      <th>dst_host_count</th>\n",
       "      <th>dst_host_srv_count</th>\n",
       "      <th>dst_host_diff_srv_rate</th>\n",
       "      <th>dst_host_same_src_port_rate</th>\n",
       "      <th>dst_host_srv_diff_host_rate</th>\n",
       "      <th>class</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.0</td>\n",
       "      <td>tcp</td>\n",
       "      <td>ftp_data</td>\n",
       "      <td>SF</td>\n",
       "      <td>491.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>150.0</td>\n",
       "      <td>25.0</td>\n",
       "      <td>0.03</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>normal</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.0</td>\n",
       "      <td>udp</td>\n",
       "      <td>other</td>\n",
       "      <td>SF</td>\n",
       "      <td>146.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.08</td>\n",
       "      <td>0.15</td>\n",
       "      <td>0.0</td>\n",
       "      <td>255.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.02</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>normal</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.0</td>\n",
       "      <td>tcp</td>\n",
       "      <td>private</td>\n",
       "      <td>S0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.05</td>\n",
       "      <td>0.07</td>\n",
       "      <td>0.0</td>\n",
       "      <td>255.0</td>\n",
       "      <td>26.0</td>\n",
       "      <td>0.05</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>anomaly</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.0</td>\n",
       "      <td>tcp</td>\n",
       "      <td>http</td>\n",
       "      <td>SF</td>\n",
       "      <td>232.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>30.0</td>\n",
       "      <td>255.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.03</td>\n",
       "      <td>0.04</td>\n",
       "      <td>normal</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.0</td>\n",
       "      <td>tcp</td>\n",
       "      <td>http</td>\n",
       "      <td>SF</td>\n",
       "      <td>199.0</td>\n",
       "      <td>420.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>255.0</td>\n",
       "      <td>255.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>normal</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 34 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   duration protocol_type   service flag  src_bytes  dst_bytes land  \\\n",
       "0       0.0           tcp  ftp_data   SF      491.0        0.0    0   \n",
       "1       0.0           udp     other   SF      146.0        0.0    0   \n",
       "2       0.0           tcp   private   S0        0.0        0.0    0   \n",
       "3       0.0           tcp      http   SF      232.0        0.0    0   \n",
       "4       0.0           tcp      http   SF      199.0      420.0    0   \n",
       "\n",
       "   wrong_fragment  urgent  hot  ...  rerror_rate same_srv_rate  diff_srv_rate  \\\n",
       "0             0.0     0.0  0.0  ...          0.0          1.00           0.00   \n",
       "1             0.0     0.0  0.0  ...          0.0          0.08           0.15   \n",
       "2             0.0     0.0  0.0  ...          0.0          0.05           0.07   \n",
       "3             0.0     0.0  0.0  ...          0.0          1.00           0.00   \n",
       "4             0.0     0.0  0.0  ...          0.0          1.00           0.00   \n",
       "\n",
       "   srv_diff_host_rate  dst_host_count  dst_host_srv_count  \\\n",
       "0                 0.0           150.0                25.0   \n",
       "1                 0.0           255.0                 1.0   \n",
       "2                 0.0           255.0                26.0   \n",
       "3                 0.0            30.0               255.0   \n",
       "4                 0.0           255.0               255.0   \n",
       "\n",
       "   dst_host_diff_srv_rate  dst_host_same_src_port_rate  \\\n",
       "0                    0.03                         0.00   \n",
       "1                    0.02                         0.00   \n",
       "2                    0.05                         0.00   \n",
       "3                    0.00                         0.03   \n",
       "4                    0.00                         0.00   \n",
       "\n",
       "   dst_host_srv_diff_host_rate    class  \n",
       "0                         0.00   normal  \n",
       "1                         0.00   normal  \n",
       "2                         0.00  anomaly  \n",
       "3                         0.04   normal  \n",
       "4                         0.00   normal  \n",
       "\n",
       "[5 rows x 34 columns]"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(f\"SHAPE  = {df.shape}\")\n",
    "df.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Almost all data preprocessing steps are complete, let's save cleaned data to file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.to_csv(\"./kdd-cleaned.csv\",index=False)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}