feat!: Implement Cloud communication reliability

[Gustrb]

Proposed changes (including videos or screenshots)

When a Workspace Registers with Cloud it gets a set of credentials it uses to generate access tokens and communicate with Cloud. Can read more about how this works in ADR3.

When a workspace owner needs to scale and deploys multiple Rocket.Chat instances to make up their deployment. Each workspace establishes a change stream request to mongo watching settings. Then updates its collection cache locally.

When a workspace goes to use an access token to talk to a cloud service it first evaluates if the token needs renewed. If it does it request a new one. Then updates a setting containing the access token. Then also another setting containing the expire.

The problem occurs when latency enters the picture. If due to either longer mongo query or network condition the other workspaces don't see both of these changes happen and update its cache. An instance can be using an expired token instead of renewing because the setting cache hasn't been updated.

We've seen many cases of this happen and typically restarting workspace can resolve.

This PR moves all credential related data to the 'workspace_credentials' collection, and stop using settings.get to fetch the configs that can be changed on multiple server instances.
This is a new PR that was created from both #32122 & #31986 in those PRs the migration was separated from the code itself, but that would end up being really hard to get merged properly, since the migration required the code, and the code assumed the migration was already ran, I'd guess we could avoid it by doing some weird PR splitting, but I am not really sure we would benefit a lot from it.

Issue(s)

Steps to test or reproduce

Create a new workspace, on registering, it should create a new record inside the rocketchat_workspace_credentials collection, containing an empty date as expirationDate, an empty string as accessToken and an empty string as scopes.
To test everything else, you must use the cloud functions, not sure how to call them from RC, but you could enter the Marketplace screen, go to paid apps, open one of them, and check if there is a new token in the database.
I would love some suggestions on how we can implement API tests for this feature, I am not sure at all if it is feasible (or if we should add later)

Further comments

This implements: https://adr.rocket.chat/0061

CONN-5

[dionisio-bot]

Looks like this PR is ready to merge! 🎉
If you have any trouble, please check the PR guidelines

[changeset-bot]

🦋 Changeset detected

Latest commit: 15f9f51

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 35 packages

Name	Type
@rocket.chat/meteor	Major
@rocket.chat/core-typings	Major
@rocket.chat/model-typings	Major
@rocket.chat/models	Major
@rocket.chat/uikit-playground	Patch
@rocket.chat/api-client	Patch
@rocket.chat/apps	Patch
@rocket.chat/core-services	Patch
@rocket.chat/cron	Patch
@rocket.chat/ddp-client	Patch
@rocket.chat/freeswitch	Patch
@rocket.chat/fuselage-ui-kit	Major
@rocket.chat/gazzodown	Major
@rocket.chat/livechat	Patch
@rocket.chat/rest-typings	Major
@rocket.chat/ui-contexts	Major
@rocket.chat/account-service	Patch
@rocket.chat/authorization-service	Patch
@rocket.chat/ddp-streamer	Patch
@rocket.chat/omnichannel-transcript	Patch
@rocket.chat/presence-service	Patch
@rocket.chat/queue-worker	Patch
@rocket.chat/stream-hub-service	Patch
@rocket.chat/license	Patch
@rocket.chat/omnichannel-services	Patch
@rocket.chat/pdf-worker	Patch
@rocket.chat/presence	Patch
rocketchat-services	Patch
@rocket.chat/instance-status	Patch
@rocket.chat/network-broker	Patch
@rocket.chat/ui-avatar	Major
@rocket.chat/ui-client	Major
@rocket.chat/ui-video-conf	Major
@rocket.chat/ui-voip	Major
@rocket.chat/web-ui-registration	Major

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 74.85%. Comparing base (f63d8e2) to head (15f9f51).
Report is 1 commits behind head on develop.

Additional details and impacted files

@@           Coverage Diff            @@
##           develop   #32856   +/-   ##
========================================
  Coverage    74.85%   74.85%           
========================================
  Files          470      470           
  Lines        20743    20743           
  Branches      5294     5294           
========================================
  Hits         15528    15528           
  Misses        4595     4595           
  Partials       620      620           

Flag	Coverage Δ
unit	74.85% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

fixed