Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CosmosClientCache changes to improve robustness #23329

Merged
merged 3 commits into from
Aug 4, 2021
Merged

CosmosClientCache changes to improve robustness #23329

merged 3 commits into from
Aug 4, 2021

Conversation

FabianMeiswinkel
Copy link
Member

  • Reducing the time stale CosmosClients are kept in the cache (to reduce memory pressure by allowing GC to eventually free up those resources). The clients are only remove from the cache - not closed - so any long running task still using them can do so without any issues - GC will just clean-up after the Spark tasks still using the clients are finished.
  • Skipping the broadcasted client cache state on Task retries (to avoid any possible issues due to stale client cache) on retries (because not all workers/executors are restarted concurrently this should be safe)

@FabianMeiswinkel
CosmosClientCache changes to improve robustness
aef94ce 
- Reducing the time stale CosmosClients are kept in the cache (to reduce memory pressure by allowing GC to eventually free up those resources). The clients are only remove from the cache - not closed - so any long running task still using them can do so without any issues - GC will just clean-up after the Spark tasks still using the clients are finished.
- Skipping the broadcasted client cache state on Task retries (to avoid any possible issues due to stale client cache) on retries (because not all workers/executors are restarted concurrently this should be safe)
@ghost ghost added the Cosmos label Aug 3, 2021
xinlian12
xinlian12 approved these changes Aug 4, 2021
View reviewed changes
Copy link
Member

@xinlian12 xinlian12 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks

@FabianMeiswinkel FabianMeiswinkel merged commit 0eec236 into Azure:main Aug 4, 2021
azure-sdk pushed a commit to azure-sdk/azure-sdk-for-java that referenced this pull request Apr 18, 2023
@Kartik-715 @cakarata
@payalguptapg @AnshulVermaa @Sandido @younghyun5756 @krishnak-msft @linugeorgeofficial @vatsan28 @vivlingaiah @mimckitt
CodeGen from PR 22885 in Azure/azure-rest-api-specs
a863ed1 
Feature/cplat 2023 03 01 (Azure#22885)

* copy last version to 2023-03-01 folder

* update version reference in the new folder

* update versions reference in the new folder

* add new readme tag with new version

* sync with updates made to 2022-11-01 version

* fix generation error

* fix duplicate enum name thru c# directive instead.

* make all the changes to this version that was made to the last version (2022-11-01) to address modelvalidation errors

* update

* Revert "update"

This reverts commit 08417d3ed412ab63c7ee7bdb712ad756ded03c97.

* Added $expand option in ListAllVMs and ListAllVMs in RG (Azure#22800)

* Added $expand option in ListAllVMs and ListAllVMs in RG

* Update virtualMachine.json

* dedicatedHost Resize feature (Azure#23268)

* dedicatedHost Resize feature

* DedicatedHost Sku renamed to size and resolved lint errors.

* reviewer comments

* httpMethod fix

* Add property for VM (Azure#22882) (Azure#23329)

Co-authored-by: payalguptapg <126145083+payalguptapg@users.noreply.github.com>

* Add Reapply for VMSS (Azure#22344)

* Add Reapply for VMSS

* Prettier fix

* Update examples

* Address review comments 01

* Use typical resourceGroupName parameter.

* Address review comments - Rename examples

* Remove <br> syntax from many descriptions in CRP swagger files (Azure#23019)

* up to computeRPCommon

* all but computeRPCommons

* computerpcommon

* vmss clean

* common clean

* vmss try n

* trying only \n

* remove \n as it messes with rest docs

* cleanup 2022-11-01 accidents

* cleanups 2023

* remove ID from Update objects that do not have ID (Azure#23078)

* update

* add identifiers

---------

Co-authored-by: Theodore Chang <theodore.l.chang@gmail.com>

* Origin/feature/cplat 2023 03 01 (Azure#23203)

* Update virtualMachine.json

Added missing property for 
Additional properties not allowed: provisionAfterExtensions
Json path: $.value[*].properties.provisionAfterExtensions

* added missing property for provisionAfterExtensions

* add locatoin in VirtualMachineScaleSetVMExtension

* fixed issue of x-ms-mutability

---------

Co-authored-by: Younghyun Kim <younghyunkim@microsoft.com>

* Add securityPostureReference (Azure#23106)

* [RestorePoints] Adding Encryption, Source Details, HyperVGeneration and WriteAccelerated for Restore Point (Azure#23303)

* Restore Point Encryption Details

* Modified descriptions and made DiskRestorePointProperties as input property

* Renaming object and adding type

* Renaming DiskRestorePointProperties

* making DiskRestorePoint.id readOnly

* Modifying reference

* Running Prettier

* Renaming EncryptionType

* Adding HyperVGeneration and WriteAcceleratorEnabled (prchin)

* Add optional parameter hibernate to vmss deallocate api (Azure#23409)

* Add optional parameter hibernate to vmss deallocate api

* Fix with prettier

* update example for vmss deallocate

* Add Spot Related Properties to VMSS PATCH

* prettier and lintDiff suppressions

* retry lintDiff suppression

* lint diff suppression retry

* Add managed identities parameters for blobs, add treatFailureAsDeploymentFailure flag for Run Command (Azure#23557)

* Add managed identities inputs for script, errorBlob, outputBlob

* Add treatFailureAsDeploymentFailure flag

* Update proximityPlacementGroup.json (Azure#23556)

* Update proximityPlacementGroup.json

* Update proximityPlacementGroup.json

* Update proximityPlacementGroup.json

---------

Co-authored-by: Theodore Chang <theodore.l.chang@gmail.com>
Co-authored-by: Kartik Gupta <31189137+Kartik-715@users.noreply.github.com>
Co-authored-by: cakarata <47228825+cakarata@users.noreply.github.com>
Co-authored-by: payalguptapg <126145083+payalguptapg@users.noreply.github.com>
Co-authored-by: Anshul Verma <88476874+AnshulVermaa@users.noreply.github.com>
Co-authored-by: Adam Sandor <adsandor@microsoft.com>
Co-authored-by: younghyun5756 <102988755+younghyun5756@users.noreply.github.com>
Co-authored-by: Younghyun Kim <younghyunkim@microsoft.com>
Co-authored-by: krishnak-msft <127885427+krishnak-msft@users.noreply.github.com>
Co-authored-by: Linu George <127192938+linugeorgeofficial@users.noreply.github.com>
Co-authored-by: vatsan28 <vatsan9228@gmail.com>
Co-authored-by: Viv Lingaiah <viv.lingaiah@gmail.com>
Co-authored-by: Micah McKittrick <32313503+mimckitt@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@xinlian12 xinlian12 xinlian12 approved these changes

@aayush3011 aayush3011 Awaiting requested review from aayush3011

@kirankumarkolli kirankumarkolli Awaiting requested review from kirankumarkolli kirankumarkolli is a code owner

@kushagraThapar kushagraThapar Awaiting requested review from kushagraThapar

@mbhaskar mbhaskar Awaiting requested review from mbhaskar

@milismsft milismsft Awaiting requested review from milismsft

@moderakh moderakh Awaiting requested review from moderakh

@simplynaveen20 simplynaveen20 Awaiting requested review from simplynaveen20

Assignees
No one assigned
Labels
Cosmos
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@FabianMeiswinkel @xinlian12