Skip to content

Remove HeadersForSearching from metadata and compute query array on the index #1410

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Sep 27, 2018

Conversation

danielmarbach
Copy link
Contributor

For ingested audit messages and error messages we created a metadata field called HeadersForSearching. That metadata field is stored in the ProcessedMessage collection as well as the FailedMessages collection. In the FailedMessages collection, we might store this field redundantly for each processing attempt. The value of that field is never directly used. It is used in the MessagesViewIndex to precompute the Query array which is then stored in the index itself. This PR removes this redundant storing from the document and modifies the MessagesViewIndex to compute the concatenated headers field and directly store it in the index.

This means we only store the headers two times (in the document and the index) per document or processing attempt instead of three times (in the document, the metadata and the index). With that change, we are saving the kbytes of data per ingested audit and error message while still being able to fully query

This will not modify existing documents but just change newly ingested once. Old documents will expire with the expiry period. Reindexing will happen on the MessagesViewIndex though.

The larger the headers the bigger the savings plus we no longer allocate the concatenated strings in ServiceControl to shuffle it to RavenDB.

Processed Message

Before

image

image

After

image

image

Failed Message

Before

image

image

After

image

image

Searching

Demonstrates how searching for custom header still works

image

image

@danielmarbach danielmarbach requested a review from a team September 11, 2018 16:17
@danielmarbach
Copy link
Contributor Author

Btw. the above optimization trick could be done also for other fields that we don't query against the metadata and thus even more storage savings could be gained

@danielmarbach danielmarbach changed the title Remove HeadersForSearching from metadata and compute query array on the index WIP Remove HeadersForSearching from metadata and compute query array on the index Sep 12, 2018
@danielmarbach danielmarbach changed the title WIP Remove HeadersForSearching from metadata and compute query array on the index Remove HeadersForSearching from metadata and compute query array on the index Sep 27, 2018
@danielmarbach danielmarbach added this to the 3.2.0 milestone Sep 27, 2018
@danielmarbach danielmarbach merged commit 575f2a8 into develop Sep 27, 2018
@danielmarbach danielmarbach deleted the save-header branch September 27, 2018 09:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants