Refactored resolvers, added string caching and xml toggling to improve memory usage #377
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Resolvers have been refactored so we can resolve specific sections independently instead of resolving everything at once. The added benefit to this is it makes string caching a bit easier and allows the cache to sit in state so multiple logs that share common data (ex. ComputerName, Source, Keywords or Category) can reuse those strings. In my testing, I have seen a reduction of about 1.5GB of memory usage on a 1GB security log that would normally use ~7GB of ram.
This above change in addition to the XML loading toggle, I have seen memory usage equivalent to 1-2x the size of the log file in my tests.
This should resolve #371 as I was able to open an 8GB (12,645,640 events) security log with XML disabled and this only utilized ~8-9GB of ram.