notebooks/1_Generate_data.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "import datetime\n",
    "import pprint\n",
    "from collections import Counter\n",
    "from pathlib import Path\n",
    "from typing import Dict, List\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import tqdm\n",
    "\n",
    "from presidio_evaluator import InputSample\n",
    "from presidio_evaluator.data_generator import PresidioDataGenerator\n",
    "from presidio_evaluator.data_generator.faker_extensions import (\n",
    "    FakerSpansResult,\n",
    "    RecordsFaker,\n",
    "    IpAddressProvider,\n",
    "    NationalityProvider,\n",
    "    OrganizationProvider,\n",
    "    UsDriverLicenseProvider,\n",
    "    AgeProvider,\n",
    "    AddressProviderNew,\n",
    "    PhoneNumberProviderNew,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Generate fake PII data using Presidio's data generator"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Presidio's data generator is based on the [Python Faker tool](https://faker.readthedocs.io/en/master/)\n",
    "and allows you to generate a synthetic dataset from sentence templates.\n",
    "It features wrappers for Faker which allows you to sample from existing sources of fake data.\n",
    "\n",
    "Example templates:\n",
    "\n",
    "> I live at {{address}}\n",
    "\n",
    "> You can email me at {{email}}. Thanks, {{first_name}}\n",
    "\n",
    "> What's your last name? It's {{last_name}}\n",
    "\n",
    "> Every time I see you falling I get down on my knees and pray\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Simple example\n",
    "This uses all the default values to generate 10 samples based on three templates"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sentence_templates = [\n",
    "    \"My name is {{name}}\",\n",
    "    \"Please send it to {{address}}\",\n",
    "    \"I just moved to {{city}} from {{country}}\",\n",
    "]\n",
    "\n",
    "\n",
    "data_generator = PresidioDataGenerator()\n",
    "fake_records = data_generator.generate_fake_data(\n",
    "    templates=sentence_templates, n_samples=10\n",
    ")\n",
    "\n",
    "fake_records = list(fake_records)\n",
    "\n",
    "# Print the spans of the first sample\n",
    "print(fake_records[0].fake)\n",
    "print(fake_records[0].spans)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Generate a full dataset\n",
    "\n",
    "In this example we customize the data generator to:\n",
    "1. Accept more types of entities (by adding more providers to Faker. see [Faker's documentation](https://faker.readthedocs.io/en/master/index.html#how-to-create-a-provider)\n",
    "2. Handle records of multiple PII entities per fake person for a more realistic dataset\n",
    "\n",
    "We then translate the generated entity types to match Presidio's, and save the new dataset in json and CONLL03 formats."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "a. Specify parameters:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "number_of_samples = 1500\n",
    "cur_time = datetime.date.today().strftime(\"%B_%d_%Y\")\n",
    "\n",
    "raw_data_path = Path(\"../presidio_evaluator/data_generator/raw_data\")\n",
    "output_file = f\"../data/generated_size_{number_of_samples}_date_{cur_time}.json\"\n",
    "output_conll = f\"../data/generated_size_{number_of_samples}_date_{cur_time}.tsv\"\n",
    "\n",
    "templates_file_path = Path(raw_data_path, \"templates.txt\").resolve()\n",
    "fake_name_generator_file = Path(raw_data_path, \"FakeNameGenerator.com_3000.csv\").resolve()\n",
    "\n",
    "lower_case_ratio = 0.05"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "b. Read [FakeNameGenerator](https://www.fakenamegenerator.com/) data (optional, extends the set of fake values)\n",
    "and create a `RecordsFaker` which returns a fake person record (with multiple values) instead of one value,\n",
    "allowing dependencies between values belonging to the same fake person\n",
    "(e.g. name = Michael Smith with the email michael.smith@gmail.com).\n",
    "\n",
    "The `fake_name_generator_file` can be downloaded from https://www.fakenamegenerator.com/order.php\n",
    "\n",
    "> Note that you can create fake records for multiple name sets, allowing you to adapt the fake data to the real data if needed. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# Read FakeNameGenerator CSV\n",
    "fake_name_generator_df = pd.read_csv(fake_name_generator_file)\n",
    "\n",
    "# Update to match existing templates\n",
    "fake_name_generator_df = PresidioDataGenerator.update_fake_name_generator_df(fake_name_generator_df)\n",
    "fake_name_generator_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "c. Create a Faker object (in this case, a `RecordsFaker`)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Create RecordsFaker (extension which handles records instead of independent values) and add additional specific providers\n",
    "fake = RecordsFaker(records=fake_name_generator_df, locale=\"en_US\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "d. Add more providers, not part of the original Faker package"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fake.add_provider(IpAddressProvider)  # Both Ipv4 and IPv6 IP addresses\n",
    "fake.add_provider(NationalityProvider)  # Read countries + nationalities from file\n",
    "fake.add_provider(OrganizationProvider)  # Read organization names from file\n",
    "fake.add_provider(UsDriverLicenseProvider)  # Read US driver license numbers from file\n",
    "fake.add_provider(AgeProvider)  # Age values (unavailable on Faker)\n",
    "fake.add_provider(AddressProviderNew)  # Extend the default address formats\n",
    "fake.add_provider(PhoneNumberProviderNew)  # Extend the default phone number formats"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "e. Create the Presidio Data Generator object and add provider aliases if the templates have a different entity name than the Faker object"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# Create Presidio Data Generator\n",
    "data_generator = PresidioDataGenerator(\n",
    "    custom_faker=fake, lower_case_ratio=lower_case_ratio\n",
    ")\n",
    "\n",
    "# Create entity aliases (e.g. if faker supports \"name\" but templates contain \"person\").\n",
    "data_generator.add_provider_alias(provider_name=\"name\", new_name=\"person\")\n",
    "data_generator.add_provider_alias(\n",
    "    provider_name=\"credit_card_number\", new_name=\"credit_card\"\n",
    ")\n",
    "data_generator.add_provider_alias(provider_name=\"date_of_birth\", new_name=\"birthday\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    },
    "scrolled": true
   },
   "source": [
    "f. Generate data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "sentence_templates = PresidioDataGenerator.read_template_file(templates_file_path)\n",
    "fake_records = data_generator.generate_fake_data(\n",
    "    templates=sentence_templates, n_samples=number_of_samples\n",
    ")\n",
    "\n",
    "fake_records = list(fake_records)\n",
    "pprint.pprint(fake_records[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Verify randomness of dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "count_per_template_id = Counter([sample.template_id for sample in fake_records])\n",
    "\n",
    "print(f\"Total: {sum(count_per_template_id.values())}\")\n",
    "print(f\"Avg # of records per template: {np.mean(list(count_per_template_id.values()))}\")\n",
    "print(f\"Median # of records per template: {np.median(list(count_per_template_id.values()))}\")\n",
    "print(f\"Std: {np.std(list(count_per_template_id.values()))}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "#### Which entities did we generate?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "count_per_entity = Counter()\n",
    "for record in fake_records:\n",
    "    count_per_entity.update(Counter([span.type for span in record.spans]))\n",
    "\n",
    "count_per_entity"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "#### Translate tags from Faker's to Presidio's (optional)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "translator = {\n",
    "    \"person\": \"PERSON\",\n",
    "    \"ip_address\": \"IP_ADDRESS\",\n",
    "    \"us_driver_license\": \"US_DRIVER_LICENSE\",\n",
    "    \"organization\": \"ORGANIZATION\",\n",
    "    \"name_female\": \"PERSON\",\n",
    "    \"address\": \"STREET_ADDRESS\",\n",
    "    \"country\": \"GPE\",\n",
    "    \"state\": \"GPE\",\n",
    "    \"credit_card_number\": \"CREDIT_CARD\",\n",
    "    \"city\": \"GPE\",\n",
    "    \"street_name\": \"STREET_ADDRESS\",\n",
    "    \"building_number\": \"STREET_ADDRESS\",\n",
    "    \"name\": \"PERSON\",\n",
    "    \"iban\": \"IBAN_CODE\",\n",
    "    \"last_name\": \"PERSON\",\n",
    "    \"last_name_male\": \"PERSON\",\n",
    "    \"last_name_female\": \"PERSON\",\n",
    "    \"first_name\": \"PERSON\",\n",
    "    \"first_name_male\": \"PERSON\",\n",
    "    \"first_name_female\": \"PERSON\",\n",
    "    \"phone_number\": \"PHONE_NUMBER\",\n",
    "    \"url\": \"DOMAIN_NAME\",\n",
    "    \"ssn\": \"US_SSN\",\n",
    "    \"email\": \"EMAIL_ADDRESS\",\n",
    "    \"date_time\": \"DATE_TIME\",\n",
    "    \"date_of_birth\": \"DATE_TIME\",\n",
    "    \"day_of_week\": \"DATE_TIME\",\n",
    "    \"year\": \"DATE_TIME\",\n",
    "    \"name_male\": \"PERSON\",\n",
    "    \"prefix_male\": \"TITLE\",\n",
    "    \"prefix_female\": \"TITLE\",\n",
    "    \"prefix\": \"TITLE\",\n",
    "    \"nationality\": \"NRP\",\n",
    "    \"nation_woman\": \"NRP\",\n",
    "    \"nation_man\": \"NRP\",\n",
    "    \"nation_plural\": \"NRP\",\n",
    "    \"first_name_nonbinary\": \"PERSON\",\n",
    "    \"postcode\": \"STREET_ADDRESS\",\n",
    "    \"secondary_address\": \"STREET_ADDRESS\",\n",
    "    \"job\": \"TITLE\",\n",
    "    \"zipcode\": \"ZIP_CODE\",\n",
    "    \"state_abbr\": \"GPE\",\n",
    "    \"age\": \"AGE\",\n",
    "}\n",
    "\n",
    "def update_entity_types(dataset:List[FakerSpansResult], entity_mapping:Dict[str,str]):\n",
    "    \"\"\"Replace entity types using a translator dictionary.\"\"\"\n",
    "\n",
    "    for sample in dataset:\n",
    "        # update entity types on spans\n",
    "        for span in sample.spans:\n",
    "            span.type = entity_mapping[span.type]\n",
    "        # update entity types on the template string\n",
    "        for key, value in entity_mapping.items():\n",
    "            sample.template = sample.template.replace(\"{{\" + key + \"}}\", \"{{\" + value + \"}}\")\n",
    "\n",
    "update_entity_types(fake_records, entity_mapping=translator)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fake_records[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Frequency of new entity types after mapping"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "\n",
    "count_per_entity_new = Counter()\n",
    "for record in fake_records:\n",
    "    for span in record.spans:\n",
    "        count_per_entity_new[span.type] += 1\n",
    "\n",
    "count_per_entity_new.most_common()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Tokenize and transform the fake samples to a list of `InputSample` objects (Common data structure for this package)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "input_samples = [\n",
    "    InputSample.from_faker_spans_result(faker_spans_result=fake_record)\n",
    "    for fake_record in tqdm.tqdm(fake_records)\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "#### Save as json"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "InputSample.to_json(dataset=input_samples, output_file=output_file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "#### Create a CONLL like data frame"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "conll = InputSample.create_conll_dataset(input_samples)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "conll.to_csv(output_conll, sep=\"\\t\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Next steps\n",
    "\n",
    "- Evaluate Presidio using this fake data. [Sample](4_Evaluate_Presidio_Analyzer.ipynb)\n",
    "- Split to train/test/validation while ensuring sentences originiating from the same template are all on the same subset. [Sample](3_Split_by_pattern_#.ipynb)\n",
    "- Conduct a small exploratory data analysis on the generated data. [Sample](2_PII_EDA.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Copyright notice:\n",
    "\n",
    "\n",
    "Data generated for evaluation was created using Fake Name Generator.\n",
    "\n",
    "Fake Name Generator identities by the [Fake Name Generator](https://www.fakenamegenerator.com/) \n",
    "are licensed under a [Creative Commons Attribution-Share Alike 3.0 United States License](http://creativecommons.org/licenses/by-sa/3.0/us/). Fake Name Generator and the Fake Name Generator logo are trademarks of Corban Works, LLC."
   ]
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "2509fbe9adc3579fd0ef23e6a2c6fb50cb745caa174aafdf017283479e60bc43"
  },
  "kernelspec": {
   "display_name": "presidio",
   "language": "python",
   "name": "presidio"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}