Remove HeadersForSearching from metadata and compute query array on the index #1410
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
For ingested audit messages and error messages we created a metadata field called
HeadersForSearching
. That metadata field is stored in theProcessedMessage
collection as well as the FailedMessages collection. In theFailedMessages
collection, we might store this field redundantly for each processing attempt. The value of that field is never directly used. It is used in theMessagesViewIndex
to precompute theQuery
array which is then stored in the index itself. This PR removes this redundant storing from the document and modifies theMessagesViewIndex
to compute the concatenated headers field and directly store it in the index.This means we only store the headers two times (in the document and the index) per document or processing attempt instead of three times (in the document, the metadata and the index). With that change, we are saving the kbytes of data per ingested audit and error message while still being able to fully query
This will not modify existing documents but just change newly ingested once. Old documents will expire with the expiry period. Reindexing will happen on the
MessagesViewIndex
though.The larger the headers the bigger the savings plus we no longer allocate the concatenated strings in ServiceControl to shuffle it to RavenDB.
Processed Message
Before
After
Failed Message
Before
After
Searching
Demonstrates how searching for custom header still works