k_anon_edu-01.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "# Introduction\n",
        "\n",
        "In this educational scenario, we will extensively explore the inner workings of the `k_anonymize` function, a tool crucial for achieving k-Anonymity in datasets. Our focus will be on comprehending the function's attributes, its underlying logic, and its pivotal role in maintaining privacy while preserving data utility.\n"
      ],
      "metadata": {
        "id": "yX7tpN3v9on3"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Step 1: Dataset Familiarization\n",
        "Commence by thoroughly understanding the dataset's description. This initial step is crucial for grasping the data's structure and identifying quasi-identifiers—attributes that could potentially reveal individual identities.\n",
        "\n",
        "```\n",
        "Age,Sex,ChestPainType,RestingBP,Cholesterol,FastingBS,RestingECG,MaxHR,ExerciseAngina,Oldpeak,ST_Slope,HeartDisease\n",
        "40,M,ATA,140,289,0,Normal,172,N,0,Up,0\n",
        "49,F,NAP,160,180,0,Normal,156,N,1,Flat,1\n",
        "37,M,ATA,130,283,0,ST,98,N,0,Up,0\n",
        "48,F,ASY,138,214,0,Normal,108,Y,1.5,Flat,1\n",
        "54,M,NAP,150,195,0,Normal,122,N,0,Up,0\n",
        "39,M,NAP,120,339,0,Normal,170,N,0,Up,0\n",
        "```\n",
        "\n",
        "## Step 2: Understanding the k_anonymize Function\n",
        "In this step, we will dive into the provided code for the `microaggregate` and `k_anonymize` functions, understanding their purpose and how they contribute to achieving k-Anonymity."
      ],
      "metadata": {
        "id": "nSx3mzvM9zOJ"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "- import csv: This line imports the \"csv\" module, which provides functionality to work with Comma-Separated Values (CSV) files. CSV files are commonly used to store tabular data, where each line of the file represents a row and the values within each line are separated by commas.\n",
        "\n",
        "- from itertools import combinations: This line imports the \"combinations\" function from the \"itertools\" module. The \"combinations\" function generates all possible combinations of a given length from a given iterable (like a list or a set).\n",
        "\n",
        "- import matplotlib.pyplot as plt: This line imports the \"pyplot\" module from the \"matplotlib\" library, which is used for creating visualizations like plots and graphs. The imported module is given an alias \"plt\" to make it easier to refer to.\n",
        "\n",
        "- import numpy as np: This line imports the \"numpy\" library, which provides support for working with arrays and matrices of numerical data. The imported library is given an alias \"np\" for convenience."
      ],
      "metadata": {
        "id": "8kJ4eMNu_acX"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "id": "FGCITL0b9eOQ"
      },
      "outputs": [],
      "source": [
        "# Import the necessary libraries\n",
        "import csv\n",
        "from itertools import combinations\n",
        "import matplotlib.pyplot as plt\n",
        "import numpy as np"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Microaggregation\n",
        "Microaggregation is a technique used in data privacy and statistical disclosure control. It's often employed when sharing or releasing data that contains sensitive or confidential information while trying to protect the individual privacy of the data subjects. Here's how the provided code works:\n",
        "\n",
        "microaggregate(data, column, k): This function takes three arguments:\n",
        "\n",
        "- data: A dataset or dataframe containing the data to be microaggregated.\n",
        "column: The column in the dataset on which microaggregation is performed.\n",
        "- k: A parameter that determines the level of aggregation. It represents the number of distinct values to be retained in the microaggregated dataset.\n",
        "- unique_vals = np.unique(data[column]): This line uses the np.unique function from the NumPy library to find the unique values in the specified column of the dataset. These unique values will be the potential representatives of the microaggregated groups.\n",
        "- aggregated_vals = {}: This initializes an empty dictionary that will store the aggregated values for each unique value.\n",
        "\n",
        "The following loop iterates through each unique value found in the specified column:\n",
        "\n",
        "- indices = np.where(data[column] == val)[0]: It finds the indices of all rows in the dataset where the column value matches the current unique value.\n",
        "- mean = np.mean(data.iloc[indices][column]): It calculates the mean of the values in the specified column for the rows corresponding to the current unique value.\n",
        "- aggregated_vals[val] = mean: It stores the calculated mean in the aggregated_vals dictionary, using the unique value as the key.\n",
        "\n",
        "The next loop iterates through each row in the dataset:\n",
        "\n",
        "- for index, row in data.iterrows(): This loop allows for iterating through both the index and the row contents of the dataset.\n",
        "- data.at[index, column] = aggregated_vals[row[column]]: It replaces the original value in the specified column with the aggregated value from the aggregated_vals dictionary, based on the value present in the current row.\n",
        "\n",
        "Finally, the modified dataset with microaggregated values is returned.\n"
      ],
      "metadata": {
        "id": "XVgNyeYE_t0-"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Define the microaggregate function\n",
        "def microaggregate(data, column, k):\n",
        "    # Find unique values in the column\n",
        "    unique_vals = np.unique(data[column])\n",
        "    # Create a dictionary to store the aggregated values\n",
        "    aggregated_vals = {}\n",
        "    for val in unique_vals:\n",
        "        # Find the indices of all rows with the same value\n",
        "        indices = np.where(data[column] == val)[0]\n",
        "        # Aggregate the values by finding the mean\n",
        "        mean = np.mean(data.iloc[indices][column])\n",
        "        # Store the aggregated value in the dictionary\n",
        "        aggregated_vals[val] = mean\n",
        "    # Replace the original values with the aggregated values\n",
        "    for index, row in data.iterrows():\n",
        "        data.at[index, column] = aggregated_vals[row[column]]\n",
        "    return data"
      ],
      "metadata": {
        "id": "Md9QPN-f-DpN"
      },
      "execution_count": 3,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# The k_anonymize function\n",
        "\n",
        "In summary, the k_anonymize function takes a CSV file, a k value, sensitive attributes, and optional generalization intervals as input. It performs k-anonymization by replacing sensitive attribute values in rows with asterisks if the rows do not meet the k-anonymity requirement. The function then writes the anonymized data to a new CSV file and returns the percentage of rows that have been successfully anonymized.\n",
        "\n",
        "Let's break down the code step by step to understand how the function works:\n",
        "\n",
        "def k_anonymize(file_path, k, sensitive_features, generalization_intervals={}):\n",
        "    # Read the CSV file\n",
        "    anon_rows = 0\n",
        "    with open(file_path, 'r') as file:\n",
        "        reader = csv.reader(file)\n",
        "        header = next(reader)\n",
        "        data = [row for row in reader]\n",
        "        row_count = len(data)\n",
        "        \n",
        "    # Find the indices of the sensitive columns\n",
        "    sensitive_indices = [header.index(feature) for feature in sensitive_features]\n",
        "The function takes several parameters:\n",
        "\n",
        "- file_path: The path to the CSV file containing the dataset.\n",
        "- k: The desired k value for k-anonymization.\n",
        "- sensitive_features: A list of column headers representing sensitive attributes.\n",
        "- generalization_intervals: A dictionary that maps column headers to generalization intervals (if any). Generalization involves replacing attribute values with broader categories to reduce identifiability.\n",
        "\n",
        "The code reads the CSV file specified by file_path using the csv.reader and extracts the header row and data rows. It also counts the total number of rows in the dataset (row_count). sensitive_indices is a list that stores the indices of the sensitive columns within the dataset, based on their headers.\n",
        "\n",
        "The code then checks if any generalization is specified for attributes. If generalization intervals are provided in the generalization_intervals dictionary, it iterates through the data and performs generalization on the corresponding attributes. Generalization replaces attribute values with ranges based on the specified intervals.\n",
        "\n",
        "    # Create a dictionary to store the count of each combination of sensitive values\n",
        "    counts = {}\n",
        "\n",
        "    # Count the occurrences of each combination of sensitive values\n",
        "    for row in data:\n",
        "        value = tuple([row[i] for i in sensitive_indices])\n",
        "        counts[value] = counts.get(value, 0) + 1\n",
        "The code creates a dictionary named counts to store the count of each combination of sensitive attribute values.\n",
        "\n",
        "It then iterates through the data rows and constructs a tuple containing the sensitive attribute values for the current row. This tuple is used as a key in the counts dictionary. The code increments the count for that specific combination of attribute values.\n",
        "\n",
        "python\n",
        "Copy code\n",
        "    # Perform the k-anonymization\n",
        "    for row in data:\n",
        "        value = tuple([row[i] for i in sensitive_indices])\n",
        "        if counts[value] < k:\n",
        "            for i in sensitive_indices:\n",
        "                row[i] = '*' * len(row[i])\n",
        "            anon_rows = anon_rows + 1\n",
        "    percentage = (anon_rows/row_count)*100\n",
        "The code then iterates through the data rows again. For each row, it checks if the count of the combination of sensitive attribute values (counts[value]) is less than the desired k value. If so, it means the row is not sufficiently anonymized, and it anonymizes the sensitive attributes by replacing their values with asterisks (*). It also increments the count of anonymized rows (anon_rows).\n",
        "\n",
        "The variable percentage is calculated to represent the percentage of rows that have been successfully anonymized.\n",
        "\n",
        "python\n",
        "Copy code\n",
        "    # Write the anonymous data to a new CSV file\n",
        "    with open('/content/sample_data/anonymised.csv', 'w', newline='') as file:\n",
        "        writer = csv.writer(file)\n",
        "        writer.writerow(header)\n",
        "        writer.writerows(data)\n",
        "    return percentage\n",
        "Finally, the code writes the anonymized data to a new CSV file and returns the calculated percentage of anonymized rows.\n",
        "\n",
        "You can run the code below:"
      ],
      "metadata": {
        "id": "JKoNtFLvAONZ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def k_anonymize(file_path, k, sensitive_features, generalization_intervals={}):\n",
        "    # Read the CSV file\n",
        "    anon_rows = 0\n",
        "    with open(file_path, 'r') as file:\n",
        "        reader = csv.reader(file)\n",
        "        header = next(reader)\n",
        "        data = [row for row in reader]\n",
        "        row_count = len(data)\n",
        "    # Find the indices of the sensitive columns\n",
        "    sensitive_indices = [header.index(feature) for feature in sensitive_features]\n",
        "\n",
        "    # Perform the generalization of attributes if necessary\n",
        "    for attribute, interval in generalization_intervals.items():\n",
        "        attribute_index = header.index(attribute)\n",
        "        for row in data:\n",
        "            row[attribute_index] = str(int(np.floor(float(row[attribute_index])/interval)) * interval) + \"-\" + str((int(np.floor(float(row[attribute_index])/interval)) + 1) * interval)\n",
        "\n",
        "    # Create a dictionary to store the count of each combination of sensitive values\n",
        "    counts = {}\n",
        "\n",
        "    # Count the occurrences of each combination of sensitive values\n",
        "    for row in data:\n",
        "        value = tuple([row[i] for i in sensitive_indices])\n",
        "        counts[value] = counts.get(value, 0) + 1\n",
        "\n",
        "    # Perform the k-anonymization\n",
        "    for row in data:\n",
        "        value = tuple([row[i] for i in sensitive_indices])\n",
        "        if counts[value] < k:\n",
        "            for i in sensitive_indices:\n",
        "                row[i] = '*' * len(row[i])\n",
        "            anon_rows = anon_rows + 1\n",
        "    percentage = (anon_rows/row_count)*100\n",
        "\n",
        "    # Write the anonymous data to a new CSV file\n",
        "    with open('/content/sample_data/anonymised.csv', 'w', newline='') as file:\n",
        "        writer = csv.writer(file)\n",
        "        writer.writerow(header)\n",
        "        writer.writerows(data)\n",
        "    return percentage"
      ],
      "metadata": {
        "id": "3BxXGmCo-FAE"
      },
      "execution_count": 4,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Step 3: Scrutinizing k_anonymize Function Attributes\n",
        "In this step, we will delve into the attributes of the k_anonymize function, understanding their roles in achieving k-Anonymity."
      ],
      "metadata": {
        "id": "JFfkM041-Oht"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Attribute explanations\n",
        "file_path = \"/content/sample_data/heart.csv\"  # Path leading to the CSV dataset\n",
        "k = 10  # Desired k-Anonymity level\n",
        "quasiidentifiers = [\"Age\",\"Cholesterol\"]  # List of attributes necessitating protection\n",
        "generalization_intervals = {\"Age\": 15}  # Optional dictionary for customized generalization levels\n",
        "\n",
        "# ... (include the provided code for experimenting with k_anonymize function calls)\n",
        "k_anonymize('/content/sample_data/heart.csv', 3, quasiidentifiers)\n",
        "\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "_NhyLe9b-SP3",
        "outputId": "799aab4e-a54d-43ca-f909-8adf89400253"
      },
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "83.2244008714597"
            ]
          },
          "metadata": {},
          "execution_count": 11
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Step 4: Practical Implementation\n",
        "Execute the provided code within the notebook. Witness the code in action by modifying attribute values. Analyze the anonymized datasets generated and the proportion of anonymized rows."
      ],
      "metadata": {
        "id": "hkRv4hR3-jiO"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Step 4: Practical Implementation\n",
        "\n",
        "# Execute the provided code to anonymize the dataset\n",
        "anon_percentage = k_anonymize(file_path, k, quasiidentifiers, generalization_intervals)\n",
        "# Display the percentage of anonymized rows\n",
        "print(\"Must be anonymized:\", anon_percentage, \"% of the dataset\")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ZsvwdKBM-ndo",
        "outputId": "37b98fda-fc9b-4906-cbb0-d4c0c94b038a"
      },
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Must be anonymized: 81.26361655773421 % of the dataset\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Step 5: Experimentation and Customization: Experiment with attribute adjustments to explore diverse scenarios:\n",
        "\n",
        "- Modify k to observe its effects on anonymity.\n",
        "Investigate different combinations of sensitive_features to discern their individual significance.\n",
        "- Customize generalization_intervals to fine-tune the degree of attribute generalization.\n",
        "- Take note of the trade-off between ensuring privacy and retaining data utility across various scenarios."
      ],
      "metadata": {
        "id": "iVvzPXfz-pUC"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "k_anonymize('/content/sample_data/heart.csv', 3, quasiidentifiers)\n",
        "anon_percentage=k_anonymize('/content/sample_data/heart.csv', 10, quasiidentifiers, generalization_intervals={\"Age\": 15})"
      ],
      "metadata": {
        "id": "JdDIBPd--sMD"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Step 5: Experimentation and Customization\n",
        "\n",
        "# Experiment with different values of k and observe their effects on anonymity\n",
        "def experiment_k_values():\n",
        "    k_values = [3, 5, 10, 15, 20]\n",
        "    for k in k_values:\n",
        "        anon_percentage = k_anonymize(file_path, k, quasiidentifiers, generalization_intervals)\n",
        "        print(f\"For k = {k}, must be anonymized: {anon_percentage:.2f}% of the dataset\")\n",
        "\n",
        "# Experiment with different combinations of sensitive features\n",
        "def experiment_quasiidentifiers():\n",
        "    scenarios = [\n",
        "        [\"Age\", \"Cholesterol\"],\n",
        "        [\"Cholesterol\", \"FastingBS\"],\n",
        "        [\"Age\", \"Cholesterol\", \"FastingBS\"]\n",
        "    ]\n",
        "    for quasiidentifiers in scenarios:\n",
        "        anon_percentage = k_anonymize(file_path, k, quasiidentifiers, generalization_intervals)\n",
        "        print(f\"Sensitive features: {', '.join(quasiidentifiers)}, must be anonymized: {anon_percentage:.2f}% of the dataset\")\n",
        "\n",
        "# Experiment with customizing generalization_intervals\n",
        "def experiment_generalization():\n",
        "    customized_generalization_intervals = {\"Age\": 10, \"Cholesterol\": 50}\n",
        "    anon_percentage = k_anonymize(file_path, k, quasiidentifiers, customized_generalization_intervals)\n",
        "    print(f\"Customized generalization, must be anonymized: {anon_percentage:.2f}% of the dataset\")\n",
        "\n",
        "# Call the experimentation functions\n",
        "experiment_k_values()\n",
        "experiment_quasiidentifiers()\n",
        "experiment_generalization()\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "nlacxyyRDxU4",
        "outputId": "52d811f5-47ec-44e7-977d-29d20a765c8f"
      },
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "For k = 3, must be anonymized: 42.05% of the dataset\n",
            "For k = 5, must be anonymized: 70.15% of the dataset\n",
            "For k = 10, must be anonymized: 81.26% of the dataset\n",
            "For k = 15, must be anonymized: 81.26% of the dataset\n",
            "For k = 20, must be anonymized: 81.26% of the dataset\n",
            "Sensitive features: Age, Cholesterol, must be anonymized: 81.26% of the dataset\n",
            "Sensitive features: Cholesterol, FastingBS, must be anonymized: 81.26% of the dataset\n",
            "Sensitive features: Age, Cholesterol, FastingBS, must be anonymized: 82.14% of the dataset\n",
            "Customized generalization, must be anonymized: 9.15% of the dataset\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Conclusion\n",
        "Summarize the key takeaways from the notebook. Emphasize how the notebook has provided an in-depth understanding of implementing k-Anonymity using the k_anonymize function. Highlight the ability to balance privacy preservation and data utility in real-world applications."
      ],
      "metadata": {
        "id": "-gVctdd--tDT"
      }
    }
  ]
}