Headers modified during message processing are not reverted for audited messages

[ramonsmits]

Describe the bug

Description

Expected behavior

Headers and body as received should be kept in their original state. "audit" headers are still added but any header modifications made in the processing pipeline should be reverted just like the message body/payload.

Actual behavior

Headers modified are not reset by audit pipeline. For example headers modified by IMutateIncomingTransportMessages leak into audit messages while the message body was correctly reverted. This caused mismatches where audit stored e.g. ContentType: application/json with a non-JSON body.

Adds SnapshotHeaders/RevertToOriginalHeadersIfNeeded to IncomingMessage, matching the existing body snapshot pattern.

Versions

Version 10.x but likely all supported versions are affected.

Steps to reproduce

1. Enable auditing
2. Add a IMutateIncomingTransportMessages that only modifies the headers
3. Review the message in the audit queue after processing and observe that the headers are not like original

Additional Information

Workarounds

Possible solutions

The following PR is an incomplete fix:

• Revert mutated headers before auditing to match body behavior #7664

It has a good comment by @danielmarbach :

• Revert mutated headers before auditing to match body behavior #7664 (comment)

It has a solution suggestion to complement RevertToOriginalBodyIfNeeded with a RevertToOriginalHeadersIfNeeded method. However, that solution uses copy dictionary that is not allocation friendly, especially when no headers are modified.

However, due to core using Dictionary in the public APIs and not IDictionary that uses a smarter "snapshot" implementation. To prevent that the following improvement is needed:

• Use IDictionary in public message and error context APIs for pipeline to reduce allocations #7738

1. Current major: Fix behavior as patch release in current major with non optimal allocations.
2. Next major: Modify API's to use IDictionary. Update revert login to use a snapshot dictionary.

Additional information

[ramonsmits]

We've reviewed this bug and added it to our backlog. It is not currently prioritized, but it may be fixed along with other future work in the repository. We will update this issue when work begins.