port and shell #74
Merged
port and shell #74
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
yuzhou03
approved these changes
May 17, 2023
upvenly
reviewed
May 23, 2023
upvenly
reviewed
May 24, 2023
upvenly
approved these changes
May 24, 2023
Ox7c000000
reviewed
May 24, 2023
| if (os.path.isfile(env_file)): | ||
| start_cmd = "cd " + dp_path + " && " + sys.executable \ | ||
| + " utils/container_manager.py -o runcmdin -c " \ | ||
| + container_name + " -d -r \"source " + env_file \ |
There was a problem hiding this comment.
start_task_cmd 里也包含了 source env_file
There was a problem hiding this comment.
对的。这些环境变量尚未在进入容器前被配置好，因此在start_task_cmd(手动)中需要先source env，然后python3 startxx.py
There was a problem hiding this comment.
已更新dev.py，对后面的start_task_cmd也做了如此处理。但start_task_cmd中存在envfile时，不需要重定向，只需要把;改成&&。因为这条命令是在容器内手动输入的，可以直接看到报错并停止运行
另外，run.py无更新内容。start_task_cmd是dev.py特有的，用于输出手动命令的。run.py中无相关逻辑
Ox7c000000
approved these changes
May 25, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
cluster_config文件中增加MASTER_PORT，默认为29501（与start_pytorch_task同步）。在某些case添加过程中遇到卡死现象，可能是PORT不可用导致的超时，但c10d默认timeout为00：30：00，导致现象看起来像卡死。可通过修改MASTER_PORT来尝试修正
run.dev添加master-port相关逻辑
run.dev在拼接source xx.env前，会检查是否存在这个文件。如不存在，则不拼接；如存在，则拼接，并将source这一句命令的stdout/stderr重定向到log_path/source_env.log.txt。并将source与python3 startxxx拼接方式改为&&。如启动task成功但在log中见不到任何输出，可查看source_env.log.txt，里面有stderr信息
同步上述修改至dev.py
已完成master-port流程测试
已完成无env文件，正确env文件，语法错误env文件测试
上述测试均在faster-rcnn stdcase中进行，以能够开始训练第一个epoch为测试成功标准