Feat/regex fallback #69
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Improve regex patterns for IP addresses, credit cards, and phone numbers - Refactor tests using parameterization for better maintainability - Add comprehensive test cases for edge cases and invalid formats - Fix validation issues with IPv6 addresses and credit card formats - Document regex pattern logic with clear comments
feat(regex): Enhance regex patterns and tests for PII detection
- Add engine parameter to TextService allowing 'regex', 'spacy', or 'auto' modes - Implement auto-fallback mechanism that tries regex first, falls back to spaCy - Add structured output option returning Span objects with position information - Create comprehensive integration tests for the new features - Update documentation in code comments, README, and CHANGELOG
feat(text-service): Add engine selection and structured output
Run weekly scheduled benchmarks Compare results against previous runs Alert on performance regressions (>10% slower)
Feat/benchmarks
Run weekly scheduled benchmarks Compare results against previous runs Alert on performance regressions (>10% slower)
Run benchmarks on pushes and pull requests
Created a GitHub Actions workflow file (.github/workflows/wheel_size.yml) to check wheel size Implemented a script (scripts/check_wheel_size.py) to verify wheel size is under 8 MB Verified locally that the wheel size is only 0.05 MB, well under our 8 MB limit Type Hints Completion Fixed all type annotation issues in the TextService class Added proper type annotations for variables and function parameters Added type checking for returned values from functions Improved code quality with better variable naming and defensive programming Documentation Improvements Added When do I need spaCy? guidance to the README Listed specific scenarios where spaCy would be beneficial despite being slower Emphasized the significant performance advantage of the regex engine CHANGELOG Updates Updated the changelog to reflect all the changes for version 4.1.0 Changed version from 4.1.0-dev to 4.1.0 Added entries for all the new features and improvements
…mentation: Set the end position of the span to match the actual length of the text (len(test_text)) Removed the PHONE span from the mock since it wasn't part of the input text Updated the test assertions to expect only one span (EMAIL) instead of two Made the test more explicit by verifying the exact properties of the span
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.