Skip to content

Multi pattern refactoring #795

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
babenek merged 4 commits into from
Dec 15, 2025
Merged

Multi pattern refactoring #795

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
e418628
MULTI rules refactoring
babenek Dec 15, 2025
b7165e5
BM test
babenek Dec 15, 2025
ad45c00
BM_test
babenek Dec 15, 2025
81aacb5
Finalize
babenek Dec 15, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
50 changes: 25 additions & 25 deletions .ci/benchmark.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
META MD5 52ca58b257084ebd44d6e812e0552686
DATA MD5 9890b532c1403b6546dde09bf66f7a41
DATA: 16995334 interested lines. MARKUP: 63710 items
META MD5 b86be0057a47695cb0050d0e187e9fc3
DATA MD5 b77c97976d58351be1aa7899c7301b66
DATA: 16995334 interested lines. MARKUP: 63736 items
FileType FileNumber ValidLines Positives Negatives
--------------- ------------ ------------ ----------- -----------
684 567150 136 695
Expand All @@ -14,7 +14,7 @@ FileType FileNumber ValidLines Positives Negatives
.backup 1 62 2
.bash 2 2158 5
.bat 5 248 2 16
.bats 15 2804 14 62
.bats 15 2804 14 64
.bazel 3 424 14
.build 2 40 1 2
.bundle 4 1512 471
Expand Down Expand Up @@ -84,11 +84,11 @@ FileType FileNumber ValidLines Positives Negatives
.ipynb 6 4804 10 10
.j 1 241 4
.j2 32 6043 7 209
.java 650 141112 478 1457
.java 650 141112 478 1455
.jenkinsfile 1 58 2 6
.jinja2 1 64 2
.js 640 530803 859 3134
.json 884 13053260 1843 10338
.json 884 13053260 1814 10391
.jsp 13 3202 1 47
.jsx 7 857 15
.jwt 1 1 2
Expand All @@ -113,13 +113,13 @@ FileType FileNumber ValidLines Positives Negatives
.lua 10 1924 3 39
.m 16 13358 22 195
.manifest 3 102 9 6
.markdown 38 5862 69 41
.markdown 38 5862 69 42
.markerb 3 12 3
.marko 1 21 2
.md 768 181113 1040 2462
.mdx 3 549 8
.mjml 1 18 1
.mjs 19 4119 91 113
.mjs 19 4119 91 116
.mk 1 5878 15
.ml 1 1856 16
.mlir 2 1596 19
Expand All @@ -139,7 +139,7 @@ FileType FileNumber ValidLines Positives Negatives
.patch 3 109384 4 25
.pbxproj 1 941 1
.pem 65 1467 64 3
.php 394 81495 167 1484
.php 394 81495 167 1487
.pl 16 14727 7 37
.pm 10 5224 1 30
.po 3 2994 11
Expand All @@ -157,14 +157,14 @@ FileType FileNumber ValidLines Positives Negatives
.pug 2 193 2
.purs 1 69 4
.pxd 1 150 2 4
.py 876 292413 748 3879
.py 876 292413 756 3880
.pyi 4 1361 10
.pyp 1 167 1
.python 1 213
.pyx 2 1094 24
.r 4 62 5 2
.rake 2 51 4
.rb 834 127133 384 3070
.rb 834 127133 383 3070
.re 1 31 1
.red 1 159 1
.release 1 13 4
Expand All @@ -179,7 +179,7 @@ FileType FileNumber ValidLines Positives Negatives
.rsa 3 3 3
.rsc 1 691 1
.rsp 18 7739 24 37
.rst 91 34736 88 345
.rst 91 34736 88 346
.rules 1 6 3
.sample 2 25 3 8
.sbt 3 570 7
Expand Down Expand Up @@ -213,7 +213,7 @@ FileType FileNumber ValidLines Positives Negatives
.travis 1 34 2 4
.ts 607 107776 266 1990
.tsx 54 7914 1 125
.txt 322 89402 5258 5785
.txt 322 89402 5258 5784
.utf8 1 77 1
.vsmdi 1 6 2 2
.vue 50 8736 1 165
Expand All @@ -225,13 +225,13 @@ FileType FileNumber ValidLines Positives Negatives
.yml 560 56585 1897 1386
.zsh 6 872 11
.zsh-theme 1 97 1
TOTAL: 11361 16995334 17154 53615
credsweeper result_cnt : 16995, lost_cnt : 0, true_cnt : 16787, false_cnt : 208
TOTAL: 11361 16995334 17132 53676
credsweeper result_cnt : 17016, lost_cnt : 0, true_cnt : 16815, false_cnt : 201
Rules Positives Negatives Reported TP FP TN FN FPR FNR ACC PRC RCL F1
------------------------------ ----------- ----------- ---------- ----- ---- ----- ---- -------- -------- -------- -------- -------- --------
API 242 4002 239 237 2 4000 5 0.000500 0.020661 0.998351 0.991632 0.979339 0.985447
AWS Client ID 205 20 196 196 0 20 9 0.000000 0.043902 0.960000 1.000000 0.956098 0.977556
AWS Multi 83 11 31 28 3 8 55 0.272727 0.662651 0.382979 0.903226 0.337349 0.491228
AWS Client ID 213 33 204 204 0 33 9 0.000000 0.042254 0.963415 1.000000 0.957746 0.978417
AWS Multi 34 66 33 33 0 66 1 0.000000 0.029412 0.990000 1.000000 0.970588 0.985075
AWS S3 Bucket 92 0 92 92 0 0 0 0.000000 1.000000 1.000000 1.000000 1.000000
Akamai Credentials 6 2 6 6 0 2 0 0.000000 0.000000 1.000000 1.000000 1.000000 1.000000
Anthropic API Key 1 0 1 1 0 0 0 0.000000 1.000000 1.000000 1.000000 1.000000
Expand All @@ -246,9 +246,9 @@ CMD Password 33 137 33 33
CMD Secret 1 18 1 1 0 18 0 0.000000 0.000000 1.000000 1.000000 1.000000 1.000000
CMD Token 6 2 6 6 0 2 0 0.000000 0.000000 1.000000 1.000000 1.000000 1.000000
CURL User Password 7 2 7 7 0 2 0 0.000000 0.000000 1.000000 1.000000 1.000000 1.000000
Credential 96 597 99 96 3 594 0 0.005025 0.000000 0.995671 0.969697 1.000000 0.984615
Credential 96 598 99 96 3 595 0 0.005017 0.000000 0.995677 0.969697 1.000000 0.984615
Docker Swarm Token 2 0 2 2 0 0 0 0.000000 1.000000 1.000000 1.000000 1.000000
Dropbox App secret 74 145 63 52 10 135 22 0.068966 0.297297 0.853881 0.838710 0.702703 0.764706
Dropbox App secret 74 145 59 52 6 139 22 0.041379 0.297297 0.872146 0.896552 0.702703 0.787879
Facebook Access Token 0 1 0 0 1 0 0.000000 1.000000
Firebase Domain 39 0 39 39 0 0 0 0.000000 1.000000 1.000000 1.000000 1.000000
Google API Key 13 0 13 13 0 0 0 0.000000 1.000000 1.000000 1.000000 1.000000
Expand All @@ -259,28 +259,28 @@ Grafana Access Policy Token 0 2 0
Grafana Provisioned API Key 7 16 7 7 0 16 0 0.000000 0.000000 1.000000 1.000000 1.000000 1.000000
Grafana Service Account Token 3 0 3 3 0 0 0 0.000000 1.000000 1.000000 1.000000 1.000000
JSON Web Token 174 61 165 165 0 61 9 0.000000 0.051724 0.961702 1.000000 0.948276 0.973451
JWK 80 0 80 80 0 0 0 0.000000 1.000000 1.000000 1.000000 1.000000
JWK 80 3 80 80 0 3 0 0.000000 0.000000 1.000000 1.000000 1.000000 1.000000
Jira / Confluence PAT token 0 4 0 0 4 0 0.000000 1.000000
Key 4213 20802 4288 4191 97 20705 22 0.004663 0.005222 0.995243 0.977379 0.994778 0.986002
Key 4225 20798 4300 4203 97 20701 22 0.004664 0.005207 0.995244 0.977442 0.994793 0.986041
MailGun API Key 8 0 8 8 0 0 0 0.000000 1.000000 1.000000 1.000000 1.000000
NKEY Seed 60 0 59 59 0 0 1 0.016667 0.983333 1.000000 0.983333 0.991597
NTLM Token 4 0 4 4 0 0 0 0.000000 1.000000 1.000000 1.000000 1.000000
Nonce 131 109 129 129 0 109 2 0.000000 0.015267 0.991667 1.000000 0.984733 0.992308
OTP / 2FA Secret 64 3 56 54 2 1 10 0.666667 0.156250 0.820896 0.964286 0.843750 0.900000
Other 0 20 0 0 20 0 0.000000 1.000000
PEM Private Key 1154 72 1154 1154 0 72 0 0.000000 0.000000 1.000000 1.000000 1.000000 1.000000
Password 2595 11364 2524 2512 12 11352 83 0.001056 0.031985 0.993194 0.995246 0.968015 0.981442
Password 2597 11362 2526 2514 12 11350 83 0.001056 0.031960 0.993194 0.995249 0.968040 0.981456
Perplexity API Key 2 0 2 2 0 0 0 0.000000 1.000000 1.000000 1.000000 1.000000
Postman Credentials 2 0 2 2 0 0 0 0.000000 1.000000 1.000000 1.000000 1.000000
SQL Password 44 14 42 42 0 14 2 0.000000 0.045455 0.965517 1.000000 0.954545 0.976744
Salesforce Credentials 6 0 6 6 0 0 0 0.000000 1.000000 1.000000 1.000000 1.000000
Salt 90 130 88 88 0 130 2 0.000000 0.022222 0.990909 1.000000 0.977778 0.988764
Secret 1520 2481 1517 1511 6 2475 9 0.002418 0.005921 0.996251 0.996045 0.994079 0.995061
Secret 1525 2476 1522 1516 6 2470 9 0.002423 0.005902 0.996251 0.996058 0.994098 0.995077
Slack Token 15 1 15 15 0 1 0 0.000000 0.000000 1.000000 1.000000 1.000000 1.000000
Stripe Credentials 2 0 2 2 0 0 0 0.000000 1.000000 1.000000 1.000000 1.000000
Tencent WeChat API App ID 47 0 47 47 0 0 0 0.000000 1.000000 1.000000 1.000000 1.000000
Token 1140 5268 1067 1062 5 5263 78 0.000949 0.068421 0.987047 0.995314 0.931579 0.962392
Twilio Credentials 30 39 30 30 0 39 0 0.000000 0.000000 1.000000 1.000000 1.000000 1.000000
URL Credentials 225 401 221 221 0 401 4 0.000000 0.017778 0.993610 1.000000 0.982222 0.991031
UUID 2517 3716 2558 2498 60 3656 19 0.016146 0.007549 0.987326 0.976544 0.992451 0.984433
17154 53615 16996 16787 208 53407 367 0.003880 0.021394 0.991875 0.987761 0.978606 0.983162
UUID 2517 3716 2554 2494 60 3656 23 0.016146 0.009138 0.986684 0.976507 0.990862 0.983632
17132 53676 17017 16815 201 53475 317 0.003745 0.018503 0.992684 0.988188 0.981497 0.984831
8 changes: 4 additions & 4 deletions .github/workflows/benchmark.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ jobs:
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2 - 2024.10.23
with:
repository: Samsung/CredData
ref: e7348eafce1e3ded950482e24fd27d026356ed81
ref: 73bee5ee1dd3557168c62fdba511d4618ac15029

- name: Markup hashing
run: |
Expand Down Expand Up @@ -87,7 +87,7 @@ jobs:
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2 - 2024.10.23
with:
repository: Samsung/CredData
ref: e7348eafce1e3ded950482e24fd27d026356ed81
ref: 73bee5ee1dd3557168c62fdba511d4618ac15029

- name: Markup hashing
run: |
Expand Down Expand Up @@ -190,7 +190,7 @@ jobs:
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2 - 2024.10.23
with:
repository: Samsung/CredData
ref: e7348eafce1e3ded950482e24fd27d026356ed81
ref: 73bee5ee1dd3557168c62fdba511d4618ac15029

- name: Markup hashing
run: |
Expand Down Expand Up @@ -378,7 +378,7 @@ jobs:
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2 - 2024.10.23
with:
repository: Samsung/CredData
ref: e7348eafce1e3ded950482e24fd27d026356ed81
ref: 73bee5ee1dd3557168c62fdba511d4618ac15029

- name: Markup hashing
run: |
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/check.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -92,7 +92,7 @@ jobs:
run: |
banner="$(python -m credsweeper --banner | head -1)"
echo "banner = '${banner}'"
if [ "CredSweeper 1.13.4 crc32:381422b0" != "${banner}" ]; then
if [ "CredSweeper 1.13.4 crc32:b4d3477f" != "${banner}" ]; then
echo "Update the check for '${banner}'"
exit 1
fi
Expand Down
1 change: 1 addition & 0 deletions credsweeper/filters/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@
from credsweeper.filters.value_not_part_encoded_check import ValueNotPartEncodedCheck
from credsweeper.filters.value_number_check import ValueNumberCheck
from credsweeper.filters.value_pattern_check import ValuePatternCheck
from credsweeper.filters.value_search_check import ValueSearchCheck
from credsweeper.filters.value_similarity_check import ValueSimilarityCheck
from credsweeper.filters.value_split_keyword_check import ValueSplitKeywordCheck
from credsweeper.filters.value_string_type_check import ValueStringTypeCheck
Expand Down
29 changes: 29 additions & 0 deletions credsweeper/filters/value_search_check.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
import re
from typing import Optional

from credsweeper.config.config import Config
from credsweeper.credentials.line_data import LineData
from credsweeper.file_handler.analysis_target import AnalysisTarget
from credsweeper.filters.filter import Filter


class ValueSearchCheck(Filter):
"""Check candidate value for a regex - useful for multi rules"""

def __init__(self, config: Optional[Config] = None, pattern: Optional[str] = None) -> None:
self.pattern: Optional[re.Pattern] = re.compile(pattern) if pattern else None

def run(self, line_data: LineData, target: AnalysisTarget) -> bool:
"""Run filter checks on received credential candidate data 'line_data'.

Args:
line_data: credential candidate data
target: multiline target from which line data was obtained

Return:
True, if need to filter candidate and False if left

"""
if self.pattern and self.pattern.search(line_data.value):
return True
return False
Loading