update error message in get_backend() more detail_ #141796

e7217 · 2024-11-29T05:04:44Z

Fixes #ISSUE_NUMBER
When attempting to reconfigure the environment without properly handling the PyTorch-related settings, you may encounter the following message.

                             │ /root/.cache/pypoetry/virtualenvs/app-rag-sample-9TtSrW0h-py3.10/lib/python3.10/site-packages/torch/distributed/distribut │               
                             │ ed_c10d.py:1215 in get_backend                                                                                            │               
                             │                                                                                                                           │               
                             │   1212 │   if _rank_not_in_group(pg):                                                                                     │               
                             │   1213 │   │   raise ValueError("Invalid process group specified")                                                        │               
                             │   1214 │   pg_store = _world.pg_map[pg] if pg in _world.pg_map else None                                                  │               
                             │ ❱ 1215 │   return Backend(not_none(pg_store)[0])                                                                          │               
                             │   1216                                                                                                                    │               
                             │   1217                                                                                                                    │               
                             │   1218 def _get_process_group_uid(pg: ProcessGroup) -> int:                                                               │               
                             │                                                                                                                           │               
                             │ /root/.cache/pypoetry/virtualenvs/app-rag-sample-9TtSrW0h-py3.10/lib/python3.10/site-packages/torch/utils/_typing_utils.p │               
                             │ y:13 in not_none                                                                                                          │               
                             │                                                                                                                           │               
                             │   10                                                                                                                      │               
                             │   11 def not_none(obj: Optional[T]) -> T:                                                                                 │               
                             │   12 │   if obj is None:                                                                                                  │               
                             │ ❱ 13 │   │   raise TypeError("Invariant encountered: value was None when it should not be")                               │               
                             │   14 │   return obj                                                                                                       │               
                             │   15                                                                                                                      │               
                             ╰───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯               
                             TypeError: Invariant encountered: value was None when it should not be                                                                      
Exception ignored in: <function Vllm.__del__ at 0x7f35f96b6dd0>

Since this message can cause confusion for multiple developers, the purpose of this PR is to suggest additional details to help clarify the situation.

cc @H-Huang @awgu @kwen2501 @wanchaol @fegin @fduwjj @wz337 @wconstab @d4l3k @c-p-i-o

pytorch-bot · 2024-11-29T05:04:48Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/141796

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit 4bb1b0d with merge base 59f14d1 ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

errant

Skylion007 · 2024-11-29T15:24:58Z

torch/distributed/distributed_c10d.py

-    pg_store = _world.pg_map[pg] if pg in _world.pg_map else None
+
+    pg_store = _world.pg_map.get(pg, None)
+    if pg_store is None:


Nice, how did this not get flagged by mypy I wonder.

@Skylion007 thank for your checking. I have resolved it.

Ah, okay this just raise the error in not_none before, and now it raises it here.

@Skylion007 Yes, that's correct. There were developers who were confused due to not knowing the cause, so I made it a bit more detailed.

e7217 · 2024-11-30T00:24:03Z

@Skylion007
Hello,

The current stage is in a waiting state, so may I kindly ask you to check it?
If any changes are required in the code, please let me know.
Thank you!

kwen2501

LGTM

e7217 · 2024-12-16T05:18:55Z

@pytorchbot merge

pytorchmergebot · 2024-12-16T05:20:58Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

pytorchmergebot · 2024-12-16T06:45:46Z

Merge failed

Reason: 1 jobs have failed, first few of them are: trunk / win-vs2019-cpu-py3 / test (default, 3, 3, lf.windows.4xlarge.nonephemeral)

Details for Dev Infra team

Raised by workflow job

pytorch-bot · 2024-12-16T08:25:31Z

You are not authorized to force merges to this repository. Please use the regular @pytorchmergebot merge command instead

e7217 · 2024-12-16T08:26:09Z

@pytorchmergebot merge

pytorchmergebot · 2024-12-16T08:27:52Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

pytorchmergebot · 2024-12-16T08:28:09Z

Merge failed

Reason: 1 jobs have failed, first few of them are: trunk / win-vs2019-cpu-py3 / test (default, 3, 3, lf.windows.4xlarge.nonephemeral)

Details for Dev Infra team

Raised by workflow job

e7217 · 2024-12-16T09:55:19Z

@pytorchmergebot merge

pytorchmergebot · 2024-12-16T09:56:58Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

pytorchmergebot · 2024-12-16T09:57:15Z

Merge failed

Reason: 1 jobs have failed, first few of them are: trunk / win-vs2019-cpu-py3 / test (default, 3, 3, lf.windows.4xlarge.nonephemeral)

Details for Dev Infra team

Raised by workflow job

e7217 · 2024-12-16T10:08:50Z

@kwen2501 Thank you for checking.

Could you please merge this PR? The CI was green in the previous iteration.

or, How can I retry it?

github-actions · 2025-02-18T00:49:23Z

Looks like this PR hasn't been updated in a while so we're going to go ahead and mark this as Stale.
Feel free to remove the Stale label if you feel this was a mistake.
If you are unable to remove the Stale label please contact a maintainer in order to do so.
If you want the bot to never mark this PR stale again, add the no-stale label.
Stale pull requests will automatically be closed after 30 days of inactivity.

kwen2501 · 2025-03-10T18:13:39Z

@pytorchbot merge -r

pytorchmergebot · 2025-03-10T18:15:08Z

@pytorchbot started a rebase job onto refs/remotes/origin/viable/strict. Check the current status here

Signed-off-by: Um Changyong <e7217@naver.com>

pytorchmergebot · 2025-03-10T18:15:12Z

Successfully rebased fix/add-error-message-more-detail onto refs/remotes/origin/viable/strict, please pull locally before adding more changes (for example, via git checkout fix/add-error-message-more-detail && git pull --rebase)

pytorchmergebot · 2025-03-10T18:16:28Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

pytorchmergebot · 2025-03-10T18:16:44Z

Merge failed

Reason: 3 mandatory check(s) failed. The first few are:

Dig deeper by viewing the failures on hud

Details for Dev Infra team

Raised by workflow job

Failing merge rule: Core Maintainers

e7217 · 2025-03-10T23:35:01Z

@kwen2501
Thank you for checking, even though it has been quite a while.
The PR is in an approved state, but for some reason, the merge has not been processed. HAHA...😂
Do you know why that might be?
Thank you once again for your follow-up, and I hope you have a great day!

kwen2501 · 2025-03-10T23:38:29Z

@pytorchbot merge

pytorchmergebot · 2025-03-10T23:40:05Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

pytorchmergebot · 2025-03-11T05:38:52Z

The merge job was canceled or timed out. This most often happen if two merge requests were issued for the same PR, or if merge job was waiting for more than 6 hours for tests to finish. In later case, please do not hesitate to reissue the merge command
For more information see pytorch-bot wiki.

e7217 · 2025-03-11T06:04:37Z

@kwen2501
First of all, I appreciate your understanding regarding my continuous requests.

Currently, even though all tests have been completed, the merge is being blocked. Is this something that needs to be addressed by the contributor or maintainer? 😮😮

I appreciate any advice you can provide.

Skylion007 · 2025-03-14T19:39:31Z

@pytorchbot merge

pytorchmergebot · 2025-03-14T19:42:14Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

pytorch-bot bot added oncall: distributed Add this issue/PR to distributed oncall triage queue release notes: distributed (c10d) release notes category labels Nov 29, 2024

pytorchbot added the open source label Nov 29, 2024

e7217 changed the title ~~update error message in get_backend() more detail~~ update error message in get_backend() more detail_ Nov 29, 2024

Skylion007 previously approved these changes Nov 29, 2024

View reviewed changes

Skylion007 reviewed Nov 29, 2024

View reviewed changes

cpuhrsch added the triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module label Dec 3, 2024

cpuhrsch requested a review from wconstab December 3, 2024 23:46

e7217 requested a review from Skylion007 December 11, 2024 17:30

kwen2501 approved these changes Dec 16, 2024

View reviewed changes

pytorch-bot bot added the ciflow/trunk Trigger trunk jobs on your pull request label Dec 16, 2024

pytorchmergebot added the merging label Dec 16, 2024

pytorchmergebot removed the merging label Dec 16, 2024

pytorchmergebot added the merging label Dec 16, 2024

pytorchmergebot removed the merging label Dec 16, 2024

pytorchmergebot added the merging label Dec 16, 2024

pytorchmergebot removed the merging label Dec 16, 2024

github-actions bot added the Stale label Feb 18, 2025

kwen2501 added no-stale and removed Stale labels Mar 10, 2025

e7217 added 2 commits March 10, 2025 18:15

update error message in get_backend() more detail

85848b6

Signed-off-by: Um Changyong <e7217@naver.com>

fix: apply lintrunner

4bb1b0d

pytorchmergebot force-pushed the fix/add-error-message-more-detail branch from e336db5 to 4bb1b0d Compare March 10, 2025 18:15

pytorch-bot bot removed the ciflow/trunk Trigger trunk jobs on your pull request label Mar 10, 2025

pytorchmergebot added the merging label Mar 10, 2025

pytorchmergebot removed the merging label Mar 10, 2025

pytorch-bot bot added the ciflow/trunk Trigger trunk jobs on your pull request label Mar 10, 2025

pytorchmergebot added the merging label Mar 10, 2025

pytorchmergebot added the Merged label Mar 14, 2025

pytorchmergebot closed this in 69aeb87 Mar 14, 2025

pytorchmergebot removed the merging label Mar 14, 2025

e7217 deleted the fix/add-error-message-more-detail branch March 15, 2025 00:56

update error message in get_backend() more detail_ #141796

update error message in get_backend() more detail_ #141796

Uh oh!

Conversation

e7217 commented Nov 29, 2024 • edited by pytorch-bot bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Nov 29, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/141796

✅ No Failures

Uh oh!

Skylion007 Nov 29, 2024

Choose a reason for hiding this comment

Uh oh!

e7217 Nov 29, 2024

Choose a reason for hiding this comment

Uh oh!

Skylion007 Nov 29, 2024

Choose a reason for hiding this comment

Uh oh!

e7217 Nov 29, 2024

Choose a reason for hiding this comment

Uh oh!

e7217 commented Nov 30, 2024

Uh oh!

kwen2501 left a comment

Choose a reason for hiding this comment

Uh oh!

e7217 commented Dec 16, 2024

Uh oh!

pytorchmergebot commented Dec 16, 2024

Merge started

Uh oh!

pytorchmergebot commented Dec 16, 2024

Merge failed

Uh oh!

pytorch-bot bot commented Dec 16, 2024

Uh oh!

e7217 commented Dec 16, 2024

Uh oh!

pytorchmergebot commented Dec 16, 2024

Merge started

Uh oh!

pytorchmergebot commented Dec 16, 2024

Merge failed

Uh oh!

e7217 commented Dec 16, 2024

Uh oh!

pytorchmergebot commented Dec 16, 2024

Merge started

Uh oh!

pytorchmergebot commented Dec 16, 2024

Merge failed

Uh oh!

e7217 commented Dec 16, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Feb 18, 2025

Uh oh!

kwen2501 commented Mar 10, 2025

Uh oh!

pytorchmergebot commented Mar 10, 2025

Uh oh!

pytorchmergebot commented Mar 10, 2025

Uh oh!

pytorchmergebot commented Mar 10, 2025

Merge started

Uh oh!

pytorchmergebot commented Mar 10, 2025

Merge failed

Uh oh!

e7217 commented Mar 10, 2025

Uh oh!

kwen2501 commented Mar 10, 2025

Uh oh!

pytorchmergebot commented Mar 10, 2025

Merge started

Uh oh!

e7217 commented Nov 29, 2024 •

edited by pytorch-bot bot

Loading

pytorch-bot bot commented Nov 29, 2024 •

edited

Loading

e7217 commented Dec 16, 2024 •

edited

Loading