Add retry mechanism to CosmosDB Database creation

[msexxeta]

Description

This is a workaround to the emulator container advertising it is ready when in fact it is not.

Fixes #9326
Fixes #7178

Checklist

• Is this feature complete?
  • Yes. Ready to ship.
  • No. Follow-up changes expected.
• Are you including unit tests for the changes and scenario tests if relevant?
  • Yes
  • No fixes existing unit test https://github.com/radical/aspire/blob/d57970b465f75fbe805bf101125ee9e4ef69882c/tests/Aspire.Hosting.Azure.Tests/AzureCosmosDBEmulatorFunctionalTests.cs#L271
• Did you add public API?
  • Yes
    • If yes, did you have an API Review for it?
      • Yes
      • No
    • Did you add <remarks /> and <code /> elements on your triple slash comments?
      • Yes
      • No
  • No
• Does the change make any security assumptions or guarantees?
  • Yes
    • If yes, have you done a threat model and had a security review?
      • Yes
      • No
  • No
• Does the change require an update in our Aspire docs?
  • Yes
    • Link to aspire-docs issue (consider using one of the following templates):
      • New (or update) doc-idea template
      • New breaking-change template
      • New diagnostic template
  • No

[msexxeta]

@dotnet-policy-service agree company="Exxeta"

[davidfowl]

Discussing with the cosmos team adding a health endpoint inside of the emulator to make this easier.

[davidfowl]

@eerhardt @sebastienros can you review this for 9.4 plz

[msexxeta]

I don't understand why Aspire.Hosting.Azure.Tests.AzureCosmosDBEmulatorFunctionalTests.VerifyWaitForOnCosmosDBEmulatorBlocksDependentResources(usePreview: False) is failing, is it possible that .OnResourceReady can be triggered before .OnConnectionStringAvailable? Or that if there are no references used, will .OnConnectionStringAvailable be skipped?

[sebastienros]

I removed the call to ReadAsync as it won't have any effect now.
We pondered if the retry was too long, but in the end if the service is really down it won't matter, and if it's up it's cached and will quickly succeed.

The only thing that is blocking now is a new test failing: Aspire.Hosting.Azure.Tests.AzureCosmosDBEmulatorFunctionalTests.VerifyWaitForOnCosmosDBEmulatorBlocksDependentResources(usePreview: False) which is probably related to the change.

Once this one passes we can merge.