Skip to content

[Fix] Fix eval stage is extremely slow #2409

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
xming521 wants to merge 1 commit into
base: main
Choose a base branch
from
+1 −0
Open

[Fix] Fix eval stage is extremely slow #2409

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions opencompass/runners/local.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -216,6 +216,7 @@ def _launch(self, task, gpu_ids, index):
param_file = f'{pwd}/tmp/{uuid_str}_params.py'

try:
object.__setattr__(task.cfg, "_format_python_code", False)
Copy link

Copilot AI Mar 5, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Using object.__setattr__() to set _format_python_code on the mmengine Config object is a fragile approach that bypasses the class's __setattr__ mechanism and relies on the internal implementation of mmengine's Config. If mmengine changes its internal attribute names or layout, this will silently fail to have any effect (without raising an error), causing the performance issue to reappear undetected.

The rest of the codebase (e.g., opencompass/cli/main.py:301 and opencompass/utils/run.py:98) passes format_python_code=False directly to Config.fromfile(). Since the task config is already a Config object at this point, a more robust alternative is to reload it with the correct flag, or to check if mmengine's Config exposes a public setter for format_python_code and use that instead.

Suggested change
object.__setattr__(task.cfg, "_format_python_code", False)
# Prefer using a public API to control python code formatting if available
if hasattr(task.cfg, 'format_python_code'):
try:
task.cfg.format_python_code = False
except Exception:
# If setting fails, fall back to default behavior
pass

Copilot uses AI. Check for mistakes.
task.cfg.dump(param_file)
Comment on lines +219 to 220
Copy link

Copilot AI Mar 5, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The fix to disable Python code formatting during config dump (line 219) is only applied in the _launch method (the non-debug path of LocalRunner), but the same slow behavior can occur in:

  1. The debug path of LocalRunner (local.py line 110), which also calls task.cfg.dump(param_file) without the fix.
  2. All other runners that call cfg.dump(param_file) without this optimization: slurm.py (line 99), dlc.py (line 127), rjob.py (line 185), volc.py (line 115), slurm_sequential.py (line 182), and local_api.py (line 200).

The fix should be applied consistently to all cfg.dump() call sites across all runners to fully address the performance issue.

Copilot uses AI. Check for mistakes.
tmpl = get_command_template(gpu_ids)
get_cmd = partial(task.get_command,
Expand Down
Loading