Template: Added a sha256 template function for obfuscating / anonymize PII data in e.g. the replace stage #2422
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Codecov Report
@@ Coverage Diff @@
## master #2422 +/- ##
==========================================
- Coverage 61.64% 61.64% -0.01%
==========================================
Files 160 160
Lines 13597 13601 +4
==========================================
+ Hits 8382 8384 +2
- Misses 4593 4594 +1
- Partials 622 623 +1
|
adityacs
reviewed
Jul 27, 2020
Comment on lines
+186
to
+190
| ```yaml | ||
| - template: | ||
| source: output | ||
| template: '{{ Sha256 .Value "salt" }}' | ||
| ``` |
There was a problem hiding this comment.
It would be good to have an example of this in replace.md as well.
cyriltovena
approved these changes
Jul 28, 2020
pkg/logentry/stages/template.go
Outdated
| @@ -32,6 +34,11 @@ var ( | |||
| "TrimPrefix": strings.TrimPrefix, | |||
| "TrimSuffix": strings.TrimSuffix, | |||
| "TrimSpace": strings.TrimSpace, | |||
| "Sha256": func(salt string, s string) string { | |||
| hasher := sha256.New() | |||
There was a problem hiding this comment.
I think this will be better https://golang.org/pkg/crypto/sha256/#Sum256 since you don’t reuse the hasher.
The reason is shorter code but also less heap allocation for high volume.
owen-d
approved these changes
Jul 28, 2020
| @@ -32,6 +34,10 @@ var ( | |||
| "TrimPrefix": strings.TrimPrefix, | |||
| "TrimSuffix": strings.TrimSuffix, | |||
| "TrimSpace": strings.TrimSpace, | |||
| "Sha256": func(salt string, s string) string { | |||
There was a problem hiding this comment.
I like this, but it'd be cool if it was multi-arity. For instance, supporting | Sha256 and | Sha256 "salt".
I can followup this PR with this change myself.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What this PR does / why we need it:
The
replacepipeline stage allows the replacement of PII values, but no obfuscating/anonymization like https://github.com/y-ken/fluent-plugin-anonymizer. By calculating the hash of a known value, you retain the possibility to query on the hashed value, so no data is lost.See this example where typical PII data like email and SSN are replaced with a hashed value
Special notes for your reviewer:
Checklist