Skip to content

Commit 7a0ff73

Browse files
authored
Update README.md
1 parent 0b3c1ba commit 7a0ff73

File tree

1 file changed

+9
-3
lines changed

1 file changed

+9
-3
lines changed

pii/README.md

Lines changed: 9 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,12 @@
1-
# PII detection and redaction for Emails, IP adresses and Secret keys
1+
# PII detection and redaction for code datasets
22

3-
We provide code to detect Emails, IP addresses and API/SSH keys in text datasets (in particular datasets of source code). We use regexes for emails and IP addresses (they are adapted from [BigScience PII pipeline](https://github.com/bigscience-workshop/data-preparation/tree/main/preprocessing/training/02_pii)). And we use [detect-secrets](https://github.com/Yelp/detect-secrets) for finding secrets keys. We additionally implement some filters on top to reduce the number of false positives. There is also some evaluation code to test the pipeline on a PII benchmark we annotated.
3+
We provide code to detect Names, Emails, IP addresses, Passwords API/SSH keys in text datasets (in particular datasets of source code).
4+
## NER approach
5+
For the **NER** model based approach go to the `ner_model` folder.
6+
7+
## Regex approach
8+
Below we explain the regex based approach to dectect Emails, IP addresses adn keys only:
9+
We use regexes for emails and IP addresses (they are adapted from [BigScience PII pipeline](https://github.com/bigscience-workshop/data-preparation/tree/main/preprocessing/training/02_pii)). And we use [detect-secrets](https://github.com/Yelp/detect-secrets) for finding secrets keys. We additionally implement some filters on top to reduce the number of false positives. There is also some evaluation code to test the pipeline on a PII benchmark we annotated.
410

511

612
We also provide the code used for training and running [StarPII](https://huggingface.co/bigcode/starpii) in `ner_model` and NER model for PII detection on: Names, Emails, Keys, Passwords & IP addresses (more details in our paper: [StarCoder: May The Source Be With You](https://drive.google.com/file/d/1cN-b9GnWtHzQRoE7M7gAEyivY0kl4BYs/view)). We provide the code (and `slurm` scripts) used for running Inference on [StarCoderData](https://huggingface.co/datasets/bigcode/starcoderdata), we were able to detect PII in ~800GB of text in 800 GPU-hours on A100 80GB. To replace secrets we used teh following tokens:
@@ -30,4 +36,4 @@ Make sure you have the `gibberish_data` folder in the same directory as the scri
3036

3137
## Notebooks
3238
* `example.ipynb` is an example notebook to show how to use the pipeline.
33-
* there are several notebooks in `notebooks` folder with some of our experiments.
39+
* there are several notebooks in `notebooks` folder with some of our experiments.

0 commit comments

Comments
 (0)