Skip to content

v2.0.1 : updated function methods, README with a Getting Started Section #2

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 8 commits into from
Feb 23, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 0 additions & 33 deletions .circleci/config.yml

This file was deleted.

4 changes: 4 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -10,4 +10,8 @@ build/
.pytest_cache/
.tox/
/src/datafog/__pycache__/
/src/datafog/pii_tools/__pycache__/
/tests/__pycache__/
/tests/scratch.py
node_modules

8 changes: 4 additions & 4 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
@@ -1,21 +1,21 @@
repos:
- repo: https://github.com/PyCQA/isort
rev: 5.12.0
rev: 5.13.2
hooks:
- id: isort

- repo: https://github.com/psf/black
rev: 22.12.0
rev: 24.2.0
hooks:
- id: black
language_version: python3

- repo: https://github.com/pycqa/flake8
rev: 6.0.0
rev: 7.0.0
hooks:
- id: flake8

- repo: https://github.com/pre-commit/mirrors-prettier
rev: v3.0.0-alpha.4
rev: v4.0.0-alpha.8
hooks:
- id: prettier
1 change: 1 addition & 0 deletions .prettierignore
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
tests/files/*
64 changes: 53 additions & 11 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,18 +7,20 @@
Scan, redact, and manage PII in your documents before they get uploaded to a Retrieval Augmented Generation (RAG) system.
</p>

[![PyPi Version](https://img.shields.io/pypi/v/datafog.svg?style=flat-square)](https://pypi.org/project/datafog/)
[![PyPI pyversions](https://img.shields.io/pypi/pyversions/datafog.svg?style=flat-square)](https://pypi.org/project/datafog/)
[![GitHub stars](https://img.shields.io/github/stars/datafog/datafog-python.svg?style=flat-square&logo=github&label=Stars&logoColor=white)](https://github.com/datafog/datafog-python)
[![PyPi downloads](https://img.shields.io/pypi/dm/datafog.svg?style=flat-square)](https://pypistats.org/packages/datafog)

<!-- [![gh-actions](https://img.shields.io/github/workflow/status/datafog/datafog-python/ci?style=flat-square)](https://github.com/datafog/datafog-python/actions?query=workflow%3Aci) -->

[![codecov](https://img.shields.io/codecov/c/github/datafog/datafog-python.svg?style=flat-square)](https://codecov.io/gh/datafog/datafog-python)
<p align="center">
<a href="https://pypi.org/project/datafog/"><img src="https://img.shields.io/pypi/v/datafog.svg?style=flat-square" alt="PyPi Version"></a>
<a href="https://pypi.org/project/datafog/"><img src="https://img.shields.io/pypi/pyversions/datafog.svg?style=flat-square" alt="PyPI pyversions"></a>
<a href="https://github.com/datafog/datafog-python"><img src="https://img.shields.io/github/stars/datafog/datafog-python.svg?style=flat-square&logo=github&label=Stars&logoColor=white" alt="GitHub stars"></a>
<a href="https://pypistats.org/packages/datafog"><img src="https://img.shields.io/pypi/dm/datafog.svg?style=flat-square" alt="PyPi downloads"></a>
</p>

<!-- [![LGTM](https://img.shields.io/lgtm/grade/python/github/datafog/datafog-python.svg?style=flat-square)](https://lgtm.com/projects/g/datafog/datafog-python) -->
<p align="center">
<a href="https://github.com/psf/black"><img src="https://img.shields.io/badge/code%20style-black-000000.svg?style=flat-square" alt="Code style: black"></a>
</p>

[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg?style=flat-square)](https://github.com/psf/black)
<p align="center">
<a href="https://codecov.io/gh/datafog/datafog-python"><img src="https://img.shields.io/codecov/c/github/datafog/datafog-python.svg?style=flat-square" alt="codecov"></a>
</p>

## Overview

Expand All @@ -36,7 +38,45 @@ DataFog can be installed via pip:
pip install datafog # python client
```

## Dev Notes
## Usage

We're going to build up functionality starting with support for the Microsoft Presidio library. If you have any custom requests that would be of benefit to the community, please let us know!

```
import requests
from datafog import PresidioEngine as presidio

# Example: Detecting PII in a String
pii_detected = presidio.scan("My name is John Doe and my email is johndoe@genai.com")
print("PII Detected:", pii_detected)

# Example: Detecting PII in a File
sample_filepath = "/Users/sidmohan/Desktop/v2.0.0/datafog-python/tests/files/input_files/sample.csv"
with open(sample_filepath, "r") as f:
original_value = f.read()
pii_detected = presidio.scan(original_value)
print("PII Detected in File:", pii_detected)

# Example: Detecting PII in a URL
sample_url = "https://gist.githubusercontent.com/sidmohan0/1aa3ec38b4e6594d3c34b113f2e0962d/raw/42e57146197be0f85a5901cd1dcdd9ad15b31bab/sotu_2023.txt"
response = requests.get(sample_url)
original_value = response.text
pii_detected = presidio.scan(original_value)
print("PII Detected in URL Content:", pii_detected)

```

Depending on your input, the output will be a list of detected PII entities:

```
PII Detected: [type: EMAIL_ADDRESS, start: 36, end: 53, score: 1.0, type: PERSON, start: 11, end: 19, score: 0.85, type: URL, start: 44, end: 53, score: 0.5]
```

## Contributing

This is an open-source project and we welcome contributions. If you have any questions, please feel free to reach out to us, join our Discord or email me directly at sid@datafog.ai.

### Dev Notes

- Clone repo
- Run 'poetry install' to install dependencies (recommend entering poetry shell for preserving dependencies)
Expand All @@ -51,7 +91,9 @@ pip install datafog # python client
To run the datafog unit tests, check out this repository and do

```

tox

```

### License
Expand Down
1 change: 0 additions & 1 deletion node_modules/.bin/prettier

This file was deleted.

22 changes: 0 additions & 22 deletions node_modules/.package-lock.json

This file was deleted.

Loading