Skip to content

DOC: separator update #801

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
babenek merged 3 commits into from
Jan 2, 2026
Merged

DOC: separator update #801

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
0b7b23d
doc separator update
babenek Jan 2, 2026
96a8e4f
BM fix
babenek Jan 2, 2026
c53a8f1
BM ref fix
babenek Jan 2, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 20 additions & 19 deletions .ci/benchmark.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
META MD5 8cf8469b772217eaa1dd9fc8c0a912de
DATA MD5 5ff49df044c012cb13bab05ee637e708
DATA: 16995334 interested lines. MARKUP: 63736 items
META MD5 346719990276f1c7ab597e7ea85f5b21
DATA MD5 d9a50a89fa4ce3c5bf3cdc5d1958ee2a
DATA: 16999171 interested lines. MARKUP: 63795 items
FileType FileNumber ValidLines Positives Negatives
--------------- ------------ ------------ ----------- -----------
684 567150 136 695
685 567300 136 695
.04 2 149 4
.1 2 641 2 10
.admx 1 26 1
Expand Down Expand Up @@ -62,7 +62,7 @@ FileType FileNumber ValidLines Positives Negatives
.gd 1 37 1
.gml 3 3075 16
.gni 3 5017 17
.go 1242 706630 1484 6137
.go 1251 709843 1494 6177
.golden 5 1168 1 50
.gradle 50 4295 8 142
.graphql 8 454 2 13
Expand Down Expand Up @@ -157,7 +157,7 @@ FileType FileNumber ValidLines Positives Negatives
.pug 2 193 2
.purs 1 69 4
.pxd 1 150 2 4
.py 876 292413 755 3881
.py 878 292875 756 3897
.pyi 4 1361 10
.pyp 1 167 1
.python 1 213
Expand Down Expand Up @@ -221,21 +221,21 @@ FileType FileNumber ValidLines Positives Negatives
.xcscheme 1 109 1
.xib 11 503 164
.xsl 1 311 1
.yaml 169 31946 207 393
.yaml 171 31958 207 395
.yml 560 56585 1897 1386
.zsh 6 872 11
.zsh-theme 1 97 1
TOTAL: 11361 16995334 17130 53678
credsweeper result_cnt : 16944, lost_cnt : 0, true_cnt : 16816, false_cnt : 128
TOTAL: 11375 16999171 17141 53736
credsweeper result_cnt : 16978, lost_cnt : 0, true_cnt : 16827, false_cnt : 151
Rules Positives Negatives Reported TP FP TN FN FPR FNR ACC PRC RCL F1
------------------------------ ----------- ----------- ---------- ----- ---- ----- ---- -------- -------- -------- -------- -------- --------
API 243 4001 238 236 2 3999 7 0.000500 0.028807 0.997879 0.991597 0.971193 0.981289
API 243 4009 242 236 6 4003 7 0.001497 0.028807 0.996943 0.975207 0.971193 0.973196
AWS Client ID 213 33 204 204 0 33 9 0.000000 0.042254 0.963415 1.000000 0.957746 0.978417
AWS Multi 34 66 33 33 0 66 1 0.000000 0.029412 0.990000 1.000000 0.970588 0.985075
AWS S3 Bucket 92 0 92 92 0 0 0 0.000000 1.000000 1.000000 1.000000 1.000000
Akamai Credentials 6 2 6 6 0 2 0 0.000000 0.000000 1.000000 1.000000 1.000000 1.000000
Anthropic API Key 1 0 1 1 0 0 0 0.000000 1.000000 1.000000 1.000000 1.000000
Auth 1165 3610 1146 1143 3 3607 22 0.000831 0.018884 0.994764 0.997382 0.981116 0.989182
Auth 1166 3616 1147 1144 3 3613 22 0.000830 0.018868 0.994772 0.997384 0.981132 0.989192
Azure Access Token 24 0 17 17 0 0 7 0.291667 0.708333 1.000000 0.708333 0.829268
BASE64 Private Key 22 4 22 22 0 4 0 0.000000 0.000000 1.000000 1.000000 1.000000 1.000000
BASE64 encoded PEM Private Key 12 0 12 12 0 0 0 0.000000 1.000000 1.000000 1.000000 1.000000
Expand All @@ -246,41 +246,42 @@ CMD Password 33 137 32 32
CMD Secret 1 18 1 1 0 18 0 0.000000 0.000000 1.000000 1.000000 1.000000 1.000000
CMD Token 6 2 6 6 0 2 0 0.000000 0.000000 1.000000 1.000000 1.000000 1.000000
CURL User Password 7 2 7 7 0 2 0 0.000000 0.000000 1.000000 1.000000 1.000000 1.000000
Credential 96 598 98 95 3 595 1 0.005017 0.010417 0.994236 0.969388 0.989583 0.979381
Credential 96 602 102 95 7 595 1 0.011628 0.010417 0.988539 0.931373 0.989583 0.959596
Docker Swarm Token 2 0 2 2 0 0 0 0.000000 1.000000 1.000000 1.000000 1.000000
Dropbox App secret 74 145 59 52 6 139 22 0.041379 0.297297 0.872146 0.896552 0.702703 0.787879
Facebook Access Token 0 1 0 0 1 0 0.000000 1.000000
Firebase Domain 39 0 39 39 0 0 0 0.000000 1.000000 1.000000 1.000000 1.000000
Github Classic Token 1 0 1 1 0 0 0 0.000000 1.000000 1.000000 1.000000 1.000000
Google API Key 13 0 13 13 0 0 0 0.000000 1.000000 1.000000 1.000000 1.000000
Google Multi 11 0 11 11 0 0 0 0.000000 1.000000 1.000000 1.000000 1.000000
Google OAuth Access Token 3 0 3 3 0 0 0 0.000000 1.000000 1.000000 1.000000 1.000000
Google OAuth Refresh Token 1 2 1 1 0 2 0 0.000000 0.000000 1.000000 1.000000 1.000000 1.000000
Grafana Access Policy Token 0 2 0 0 2 0 0.000000 1.000000
Grafana Provisioned API Key 7 16 7 7 0 16 0 0.000000 0.000000 1.000000 1.000000 1.000000 1.000000
Grafana Service Account Token 3 0 3 3 0 0 0 0.000000 1.000000 1.000000 1.000000 1.000000
JSON Web Token 174 61 165 165 0 61 9 0.000000 0.051724 0.961702 1.000000 0.948276 0.973451
JSON Web Token 175 61 166 166 0 61 9 0.000000 0.051429 0.961864 1.000000 0.948571 0.973607
JWK 80 3 80 80 0 3 0 0.000000 0.000000 1.000000 1.000000 1.000000 1.000000
Jira / Confluence PAT token 0 4 0 0 4 0 0.000000 1.000000
Key 4225 20798 4247 4210 37 20761 15 0.001779 0.003550 0.997922 0.991288 0.996450 0.993862
Key 4225 20804 4249 4210 39 20765 15 0.001875 0.003550 0.997843 0.990821 0.996450 0.993628
MailGun API Key 8 0 8 8 0 0 0 0.000000 1.000000 1.000000 1.000000 1.000000
NKEY Seed 60 0 59 59 0 0 1 0.016667 0.983333 1.000000 0.983333 0.991597
NTLM Token 4 0 4 4 0 0 0 0.000000 1.000000 1.000000 1.000000 1.000000
Nonce 131 109 128 127 1 108 4 0.009174 0.030534 0.979167 0.992188 0.969466 0.980695
OTP / 2FA Secret 64 3 56 54 2 1 10 0.666667 0.156250 0.820896 0.964286 0.843750 0.900000
Other 0 20 0 0 20 0 0.000000 1.000000
PEM Private Key 1154 72 1154 1154 0 72 0 0.000000 0.000000 1.000000 1.000000 1.000000 1.000000
Password 2595 11364 2525 2517 8 11356 78 0.000704 0.030058 0.993839 0.996832 0.969942 0.983203
PEM Private Key 1157 72 1157 1157 0 72 0 0.000000 0.000000 1.000000 1.000000 1.000000 1.000000
Password 2595 11366 2527 2517 10 11356 78 0.000880 0.030058 0.993697 0.996043 0.969942 0.982819
Perplexity API Key 2 0 2 2 0 0 0 0.000000 1.000000 1.000000 1.000000 1.000000
Postman Credentials 2 0 2 2 0 0 0 0.000000 1.000000 1.000000 1.000000 1.000000
SQL Password 44 14 41 40 1 13 4 0.071429 0.090909 0.913793 0.975610 0.909091 0.941176
Salesforce Credentials 6 0 6 6 0 0 0 0.000000 1.000000 1.000000 1.000000 1.000000
Salt 90 130 88 88 0 130 2 0.000000 0.022222 0.990909 1.000000 0.977778 0.988764
Secret 1525 2476 1511 1510 1 2475 15 0.000404 0.009836 0.996001 0.999338 0.990164 0.994730
Secret 1525 2492 1519 1510 9 2483 15 0.003612 0.009836 0.994025 0.994075 0.990164 0.992116
Slack Token 15 1 15 15 0 1 0 0.000000 0.000000 1.000000 1.000000 1.000000 1.000000
Stripe Credentials 2 0 2 2 0 0 0 0.000000 1.000000 1.000000 1.000000 1.000000
Tencent WeChat API App ID 47 0 47 47 0 0 0 0.000000 1.000000 1.000000 1.000000 1.000000
Token 1139 5269 1064 1061 3 5266 78 0.000569 0.068481 0.987360 0.997180 0.931519 0.963232
Token 1144 5285 1072 1066 6 5279 78 0.001135 0.068182 0.986934 0.994403 0.931818 0.962094
Twilio Credentials 30 39 30 30 0 39 0 0.000000 0.000000 1.000000 1.000000 1.000000 1.000000
URL Credentials 225 401 221 220 1 400 5 0.002494 0.022222 0.990415 0.995475 0.977778 0.986547
UUID 2517 3716 2554 2494 60 3656 23 0.016146 0.009138 0.986684 0.976507 0.990862 0.983632
17130 53678 16945 16816 128 53550 314 0.002385 0.018330 0.993758 0.992446 0.981670 0.987028
17141 53736 16979 16827 151 53585 314 0.002810 0.018319 0.993439 0.991106 0.981681 0.986371
8 changes: 4 additions & 4 deletions .github/workflows/benchmark.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ jobs:
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2 - 2024.10.23
with:
repository: Samsung/CredData
ref: 49e136abfeb5fa7498139d34a45c4e97ffc9854e
ref: ac329e6de7d5c765c61f1abce5851ea3d4134131

- name: Markup hashing
run: |
Expand Down Expand Up @@ -87,7 +87,7 @@ jobs:
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2 - 2024.10.23
with:
repository: Samsung/CredData
ref: 49e136abfeb5fa7498139d34a45c4e97ffc9854e
ref: ac329e6de7d5c765c61f1abce5851ea3d4134131

- name: Markup hashing
run: |
Expand Down Expand Up @@ -190,7 +190,7 @@ jobs:
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2 - 2024.10.23
with:
repository: Samsung/CredData
ref: 49e136abfeb5fa7498139d34a45c4e97ffc9854e
ref: ac329e6de7d5c765c61f1abce5851ea3d4134131

- name: Markup hashing
run: |
Expand Down Expand Up @@ -378,7 +378,7 @@ jobs:
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2 - 2024.10.23
with:
repository: Samsung/CredData
ref: 49e136abfeb5fa7498139d34a45c4e97ffc9854e
ref: ac329e6de7d5c765c61f1abce5851ea3d4134131

- name: Markup hashing
run: |
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/check.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ jobs:
- name: Check ml_config.json and ml_model.onnx integrity
if: ${{ always() && steps.code_checkout.conclusion == 'success' }}
run: |
md5sum --binary credsweeper/ml_model/ml_config.json | grep b2f99860e0a630e91f91392584b6f533
md5sum --binary credsweeper/ml_model/ml_config.json | grep 326a4f7543f5cee73e92e05289362664
md5sum --binary credsweeper/ml_model/ml_model.onnx | grep a3054b55c88dd63a168f8e1fe27bbbdf

# # # line ending
Expand Down Expand Up @@ -92,7 +92,7 @@ jobs:
run: |
banner="$(python -m credsweeper --banner | head -1)"
echo "banner = '${banner}'"
if [ "CredSweeper 1.14.2 crc32:36ab773c" != "${banner}" ]; then
if [ "CredSweeper 1.14.2 crc32:a2e9bad4" != "${banner}" ]; then
echo "Update the check for '${banner}'"
exit 1
fi
Expand Down
6 changes: 3 additions & 3 deletions credsweeper/ml_model/ml_config.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -79,7 +79,7 @@
"type": "SearchInAttribute",
"comment": "ASN1 prefix for PEM keys",
"kwargs": {
"pattern": "MII",
"pattern": "\\b(MII|LS0t)",
"attribute": "value"
}
},
Expand Down Expand Up @@ -127,7 +127,7 @@
"type": "SearchInAttribute",
"comment": "VariableNotAllowedNameCheck - hash mentioned",
"kwargs": {
"pattern": "(?i:(md5|sha[_-]?(224|256|384|512)))",
"pattern": "(?i:( h1$|md5|sha[_-]?(224|256|384|512)))",
"attribute": "variable"
}
},
Expand Down Expand Up @@ -732,4 +732,4 @@
}
}
]
}
}
2 changes: 1 addition & 1 deletion credsweeper/rules/config.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@
confidence: weak
type: pattern
values:
- (?P<wrap>[\"'`(])?\s*(?P<variable>(\w*(?i:(?<!by)passw?o?r?d?s?(?!e[dns]|ing|ion|age)|pwd?\b|\bp/w\b|token|secret|key|credential)\w*|비밀번호|비번|패스워드|키|암호화?|토큰))[\"'`]*(\s+(?i:is|are|was|were)(\s*[:-])?\s+|\s*(설정은|[=:!]{1,3})\s*)(?P<quote>[\"'`]{1,6})?(?P<value>(?(quote)(?(wrap)[^\"'`)]{4,80}|[^\"'`]{4,80})|(?(wrap)[^\"'`)]{4,80}|\S{4,80})))
- (?P<wrap>[\"'`(])?\s*(?P<variable>(\w*(?i:(?<!by)passw?o?r?d?s?(?!e[dns]|ing|ion|age)|pwd?\b|\bp/w\b|token|secret|key|credential)\w*|비밀번호|비번|패스워드|키|암호화?|토큰))[\"'`]*(\s+(?i:is|are|was|were)(\s*[:-])?\s+|\s*(?P<separator>설정은|:=|:(?!:)|=(>|&gt;|(\\\\*u00|%)26gt;)|!==|!=|===|==|=~|=|%3[Dd])\s*)(?P<quote>[\"'`]{1,6})?(?P<value>(?(quote)(?(wrap)[^\"'`)]{4,80}|[^\"'`]{4,80})|(?(wrap)[^\"'`)]{4,80}|\S{4,80})))
filter_type:
- ValueAllowlistCheck
- LineGitBinaryCheck
Expand Down
8 changes: 4 additions & 4 deletions tests/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,16 +7,16 @@
NEGLIGIBLE_ML_THRESHOLD = 0.0001

# with option --doc & NEGLIGIBLE_ML_THRESHOLD
SAMPLES_IN_DOC = 881
SAMPLES_IN_DOC = 884

# credentials count after scan without filters and ML validations
SAMPLES_REGEX_COUNT = 649
SAMPLES_REGEX_COUNT = 650

# credentials count after scan with filters and without ML validation
SAMPLES_FILTERED_COUNT = 538
SAMPLES_FILTERED_COUNT = 539

# credentials count after default post-processing
SAMPLES_POST_CRED_COUNT = 491
SAMPLES_POST_CRED_COUNT = 492

# archived credentials that are not found without --depth
SAMPLES_IN_DEEP_1 = SAMPLES_POST_CRED_COUNT + 138
Expand Down
23 changes: 22 additions & 1 deletion tests/data/depth_3_pedantic.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1701,6 +1701,27 @@
}
]
},
{
"rule": "Password",
"severity": "high",
"confidence": "moderate",
"ml_probability": 0.996,
"line_data_list": [
{
"line": "password:!Dd5G4s9",
"line_num": 3,
"path": "./tests/samples/doc_credentials_p",
"info": "FILE:./tests/samples/doc_credentials_p|RAW",
"variable": "password",
"variable_start": 0,
"variable_end": 8,
"value": "!Dd5G4s9",
"value_start": 9,
"value_end": 17,
"entropy": 3.0
}
]
},
{
"rule": "Secret",
"severity": "medium",
Expand Down Expand Up @@ -7473,7 +7494,7 @@
"rule": "Key",
"severity": "high",
"confidence": "moderate",
"ml_probability": 0.999,
"ml_probability": 0.993,
"line_data_list": [
{
"line": "pkey = (\"LS0tLS1CRUdJTiBQUklWQVRFIENDcUdTTTQ5QXdFSEJHMHdhd0lCQVFRZ0ViVn\"",
Expand Down
Loading