[filebeat][google workspace] - Added canonical sorting over fingerprint keys array to maintain key order

[ShourieG]

Type of change

• Bug

Proposed commit message

The Google Workspace module has had a long standing issue of generating duplicate entries for events. This is an attempt to address that issue by introducing a canonical order to the fingerprint keys. It is possible that over successive responses, the same event object could possibly be canonically unordered. This is hinted by issues like this where ingested events have unique fingerprints for the same event leading to the duplication of events in elasticsearch.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Disruptive User Impact

User might initially get more duplicates after updating, but this will subside after sometime as the ingestion process self corrects. This is because the sorting will impact the fingerprint of events to an extent in the initial period.

Author's Checklist

• [ ]

How to test this PR locally

Related issues

• Relates Duplicated Google Workspace log entries by Filebeat #39859, #44794

Use cases

Screenshots

Logs

[mergify]

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @ShourieG? 🙏.
For such, you'll need to label your PR with:

• The upcoming major version of the Elastic Stack
• The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-v8./d.0 is the label to automatically backport to the 8./d branch. /d is the digit

[efd6]

I have doubts that this will fix the issue if it is related to #39859; in that issue we see that there are missing fields in the resulting document; there is no json.id.uniqueQualifier/event.id, json.id.applicationName/event.provider or json.id.customerId/organization.id which are all (in the first incarnation) used in the fingerprinting processor. These do all show up in the event.original, so we know that they have been received. Looking at the processors' implementations, I don't see any obvious reason why fields would be being lost.

What I do find confusing is that in the documents shown in the issue, we still see the json. … fields, but these should have been removed here (@rlevytskyi do you have local changes to your configuration that would have stopped the ingest pipeline being run?)

[rlevytskyi]

do you have local changes to your configuration that would have stopped the ingest pipeline being run

Absolutely no. Events got fetched by Filebeat, then passed to Logstash, then passed to Opensearch.

[efd6]

@rlevytskyi Thanks.

This complicates the issue and might explain the confounding behaviour.

then passed to Logstash

[efd6]

Approving because I don't see this causing any harm, but I still have doubt that it will fix the issue. Please get a second opinion.

[andrewkroh]

This whole section looks like it is working around limitations in the beat fingerprint processor. And it's not entirely overcoming those limitations (hashing arrays and objects). I think the would be reasonable to enhance the fingerprint processor to accommodate this usage (the ES fingerprint processor does this). Or move this _id processing into ingest node.

But I would consider fingerprinting on the JSON string that is contained in message if the full contents are suitable for a fingerprint (i.e. each time an event is requested the api returns the same JSON).

[ShourieG]

I'm a bit worried atm to change the fingerprint processor behaviour since changing it there would impact all modules utilising the processor and could possibly cause an initial wave of duplicates all across the board since the sorting will impact the fingerprint. I'll bring it up in the team sync once to discuss this and then make the change. For now I'm merging this change(otherwise it will go past FF) and we'll see and observe if this impacts duplicate creation in any way.