[core] cache find_gcs_addresses by rueian · Pull Request #61065 · ray-project/ray

rueian · 2026-02-14T06:37:46Z

Description

Executing the find_gcs_addresses function on a head node, where the RAY_ADDRESS env is not set, will scan the user's process list and find GCS addresses from each of the command line arguments in the list. The process is super expensive, as shown in the screenshot below. It dominates the detect function in Ray Data's hanging detector.

This PR caches the result of find_gcs_addresses if any.

Related issues

Link related issues: "Fixes #1234", "Closes #1234", or "Related to #1234".

Additional information

Optional: Add implementation details, API changes, usage examples, screenshots, etc.

gemini-code-assist

Code Review

This pull request introduces caching for the find_gcs_addresses function using @functools.cache. While this is a good performance optimization, it introduces the risk of returning stale data if GCS processes change during the lifetime of the client process. This could lead to connection issues in dynamic environments where clusters are started and stopped. My review comment details this concern and suggests potential mitigations.

python/ray/_private/services.py

github-actions · 2026-03-01T00:57:28Z

This pull request has been automatically marked as stale because it has not had
any activity for 14 days. It will be closed in another 14 days if no further activity occurs.
Thank you for your contributions.

You can always ask for help on our discussion forum or Ray's public slack channel.

If you'd like to keep this open, just leave any comment, and the stale label will be removed.

Signed-off-by: Rueian Huang <rueiancsie@gmail.com>

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b6a6be1592

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-03-16T16:48:34Z

python/ray/_private/services.py

+@cache
+def _cached_find_gcs_addresses():
+    return frozenset(_find_address_from_flag("--gcs-address"))


Invalidate cached GCS addresses after external Ray stop

Caching find_gcs_addresses() at module scope makes the first non-empty process scan persist for the lifetime of the Python process, but only a subset of call paths clear it. I checked callers like canonicalize_bootstrap_address_or_die used by python/ray/dashboard/utils.py and python/ray/_private/internal_api.py; those paths can run after a cluster is stopped/restarted externally (e.g., ray stop in another process), and they will continue to reuse the stale cached address instead of re-scanning, causing auto-discovery to target a dead GCS endpoint.

Useful? React with 👍 / 👎.

codope · 2026-03-16T17:10:06Z

python/ray/_private/services.py

+    addresses = _cached_find_gcs_addresses()
+    if not addresses:
+        _cached_find_gcs_addresses.cache_clear()
+    return set(addresses)


This is not atomic right? Between the call and the cache_clear, is it possible another thread could read the (now-stale empty) cached value?

Also, now there is a new set() from the cached frozenset on every call, which is intentional (to avoid callers mutating the cached data), but given this is on the hot path, it's worth confirming the set() copy cost is negligible compared to the process scan.

This is not atomic right? Between the call and the cache_clear, is it possible another thread could read the (now-stale empty) cached value?

Correct. It is possible that another thread reads an empty cached value and clears the cache again. Upon checking, I think that currently there is no such concurrent thread that uses the function, but adding a thread lock might still be a good idea.

it's worth confirming the set() copy cost is negligible compared to the process scan.

A copy is currently necessary because the caller will alter the result. But we can also make the caller not alter the result.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 7640716d26

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-03-16T18:39:04Z

python/ray/_private/services.py

+    with _find_gcs_addresses_lock:
+        addresses = _cached_find_gcs_addresses()
+        if not addresses:
+            _cached_find_gcs_addresses.cache_clear()


Add invalidation strategy for cached GCS address scans

Caching find_gcs_addresses() indefinitely means any non-empty process-scan result can outlive the actual cluster lifecycle, and only a few paths now clear it manually (worker.py/cluster_utils.py). Callers that do not clear this cache (for example scripts.start's preflight check in python/ray/scripts/scripts.py and canonicalize_bootstrap_address_or_die users) can keep using dead addresses after a local ray stop/restart in another process, causing false "already running" or failed auto-connect behavior until interpreter restart. Fresh evidence beyond the existing comment is that scripts.start still consults find_gcs_addresses() directly without any corresponding cache clear.

Useful? React with 👍 / 👎.

cursor

Cursor Bugbot has reviewed your changes and found 1 potential issue.

python/ray/_private/services.py

Signed-off-by: Rueian Huang <rueiancsie@gmail.com>

gemini-code-assist bot reviewed Feb 14, 2026

View reviewed changes

python/ray/_private/services.py Outdated Show resolved Hide resolved

rueian force-pushed the cache-find-gcs-address branch from d454587 to b111d45 Compare February 14, 2026 06:50

rueian changed the title ~~[core] cache find_gcs_addresses~~ [core] cache get_address_for_submission_client Feb 14, 2026

rueian force-pushed the cache-find-gcs-address branch from b111d45 to 3b58b5f Compare February 14, 2026 07:09

rueian added core Issues that should be addressed in Ray Core go add ONLY when ready to merge, run all tests labels Feb 14, 2026

rueian force-pushed the cache-find-gcs-address branch from 3b58b5f to e00ae0e Compare February 14, 2026 17:46

github-actions bot added the stale The issue is stale. It will be closed within 7 days unless there are further conversation label Mar 1, 2026

rueian force-pushed the cache-find-gcs-address branch from e00ae0e to 5aa9f3f Compare March 12, 2026 00:13

github-actions bot added unstale A PR that has been marked unstale. It will not get marked stale again if this label is on it. and removed stale The issue is stale. It will be closed within 7 days unless there are further conversation labels Mar 12, 2026

rueian force-pushed the cache-find-gcs-address branch 7 times, most recently from fef7c0c to a31c8bb Compare March 14, 2026 17:37

[core] cache get_address_for_submission_client

b6a6be1

Signed-off-by: Rueian Huang <rueiancsie@gmail.com>

rueian force-pushed the cache-find-gcs-address branch from a31c8bb to b6a6be1 Compare March 14, 2026 22:56

rueian changed the title ~~[core] cache get_address_for_submission_client~~ [core] cache find_gcs_addresses Mar 16, 2026

rueian marked this pull request as ready for review March 16, 2026 16:42

rueian requested a review from a team as a code owner March 16, 2026 16:42

chatgpt-codex-connector bot reviewed Mar 16, 2026

View reviewed changes

codope reviewed Mar 16, 2026

View reviewed changes

chatgpt-codex-connector bot reviewed Mar 16, 2026

View reviewed changes

cursor bot reviewed Mar 16, 2026

View reviewed changes

python/ray/_private/services.py Show resolved Hide resolved

address comment

546a0de

Signed-off-by: Rueian Huang <rueiancsie@gmail.com>

rueian force-pushed the cache-find-gcs-address branch from 7640716 to 546a0de Compare March 16, 2026 20:15

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[core] cache find_gcs_addresses#61065

[core] cache find_gcs_addresses#61065
rueian wants to merge 2 commits intoray-project:masterfrom
rueian:cache-find-gcs-address

rueian commented Feb 14, 2026 •

edited

Loading

Uh oh!

gemini-code-assist bot left a comment

Uh oh!

Uh oh!

github-actions bot commented Mar 1, 2026

Uh oh!

chatgpt-codex-connector bot left a comment

Uh oh!

chatgpt-codex-connector bot Mar 16, 2026

Uh oh!

codope Mar 16, 2026

Uh oh!

rueian Mar 16, 2026

Uh oh!

chatgpt-codex-connector bot left a comment

Uh oh!

chatgpt-codex-connector bot Mar 16, 2026

Uh oh!

cursor bot left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

rueian commented Feb 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related issues

Additional information

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

github-actions bot commented Mar 1, 2026

Uh oh!

chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector bot Mar 16, 2026

Choose a reason for hiding this comment

Uh oh!

codope Mar 16, 2026

Choose a reason for hiding this comment

Uh oh!

rueian Mar 16, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector bot Mar 16, 2026

Choose a reason for hiding this comment

Uh oh!

cursor bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

rueian commented Feb 14, 2026 •

edited

Loading