Track processed events in Azure provider

[MarcAstr0]

Description

The changes in this PR aim to correct a behavior noticed in the Azure provider, where the Cosmos DB change feed would get triggered more than once for the same event, resulting in the same event being processed twice. This behavior has been seen to happen during load-balancing scenarios.

To correct this duplicate-event behavior, a new processed events container is added to Cosmos DB to store the IDs of events that the event processor has processed. The processor will query this container for the event IDs and verify the event has not been processed (i.e., it can't find its ID in this new processed-events container). If an event with the same ID is found, to avoid duplication, the event processor will ignore it and log a warning indicating the duplication.

The IDs store in the processed events container have a time-to-live (TTL) and will get cleared automatically.

Changes

• Modifies framework-provider-azure-infrastructure package adding the new container for processed events.
• Adds a configurable processedEventsTtl to BoosterConfig for setting the TTL of the event IDs in the processed events container (defaults to 300 seconds, i.e., 5 minutes).
• Makes the necessary changes to the BoosterEventProcessor in the framework-core package for querying and storing the processed event IDs.
• Modifies the adapters in the framework-provider-azure package for interacting with the processed events container.
• For backwards compatibility, adds dummy functions to the (now deprecated) AWS and the local provider packages that will skip any verification for duplicate events since this behavior hasn't been reported in these providers.

Checks

• Project Builds
• Project passes tests and checks
• Updated documentation accordingly

[what-the-diff]

PR Summary

• Tracking of Processed Events
  Added a new JSON file to keep track of processed events in Azure. This ensures the repetition of events is avoided, enhancing performance and efficiency.

• Update in the Core Framework
  Added new functionalities in the core framework which allow for filtering and storing of unprocessed events. These changes ensure that only new events are dispatched to event handlers.

• Addition of Event Storage Utility
  Added new methods in the event storage service to handle storing and searching for processed events. This facilitates efficient event tracking and retrieval.

• New Tests for Event Processing
  Incorporated a new test case to verify if already processed events are filtered before dispatch to event handlers, enhancing the accuracy of event dispatches.

• AWS and Azure Infrastructure Update
  For both providers, added dummy methods that return an empty array representing processed events. These methods ensure compatibility across different providers.

• Azure Infrastructure Enhancement
  By creating a container for storing processed events, the improved Azure infrastructure facilitates the tracking and retrieval of processed events.

• Improved Event Adapters
  For both AWS and Azure, new methods were introduced for storing and fetching processed events. This results in better management of events and ensures processed events can be easily retrieved.

• Configurations and Interface Enhancements
  Several new methods, interfaces, and properties have been added to different files. These additions provide new features such as processed events storage and search, enhancing the flexibility and functionality of the booster configuration class.

• Minor Changes
  Small changes have been made to some files, assuring a better and more refined coding structure.

[MarcAstr0]

/integration sha=cffd25137016b4b4eae2032990dfc81c8e8fee6f

[NickSeagull]

/integration sha=cffd25137016b4b4eae2032990dfc81c8e8fee6f

[github-actions]

⌛ Integration tests are running...

Check their status here 👈

[github-actions]

❌ Oh no! Integration tests have failed

[MarcAstr0]

/integration sha=cffd25137016b4b4eae2032990dfc81c8e8fee6f

[github-actions]

⌛ Integration tests are running...

Check their status here 👈

[github-actions]

❌ Oh no! Integration tests have failed

[MarcAstr0]

/integration sha=79aacd308959f34dcad0106d20bc9cb3949a3f9d

[github-actions]

⌛ Integration tests are running...

Check their status here 👈

[github-actions]

✅ Integration tests have finished successfully!

[alvaroloes]

I left some nitpicking comments and one a bit more important about what to return when any other error happens while storing the dispatched event

[MarcAstr0]

@alvaroloes all of your comments have been addressed in commit 8b06598 😉 👍

[MarcAstr0]

/integration sha=8b06598a412b227c4b93d4152d4ff6bba64a3c5a

[github-actions]

⌛ Integration tests are running...

Check their status here 👈

[github-actions]

✅ Integration tests have finished successfully!

[alvaroloes]

Approved, but please take a look at the error "bubble up" comment and the login thing, so that it is easier to debug in prod.