Refactor find cached script logic in script automation #750

sujik18 · 2025-12-28T05:05:58Z

🧾 PR Checklist

Target branch is dev

📌 Note: PRs must be raised against dev. Do not commit directly to main.

📁 File Hygiene & Output Handling

No unintended files (e.g., logs, cache, temp files, pycache, output folders) are committed

📝 Comments & Communication

Proper inline comments are added to explain important or non-obvious changes
PR title and description clearly state what the PR does and why
Related issues (if any) are properly referenced (Fixes #, Related to #, etc.)

🛡️ Safety & Security

No secrets or credentials are committed
Paths, shell commands, and environment handling are safe and portable

Modularized find_cached_script function and have moved few additional helper functions to script_utils.py
Logs:

(mlcflow) sujith@ideapad-g3:~$ mlc run script --tags=detect,os -j -v
[2025-12-28 10:29:42,465 index.py:32 DEBUG] - Repos path for Index: /home/sujith/MLC/repos
[2025-12-28 10:29:42,468 index.py:174 DEBUG] - index_script.json exists. Skipping forced rebuild.
[2025-12-28 10:29:42,468 index.py:184 DEBUG] - Current repos.json mtime: 1766834865.1588597
[2025-12-28 10:29:42,468 index.py:185 DEBUG] - Old repos.json mtime: 1766834865.1588597
[2025-12-28 10:29:42,468 index.py:204 DEBUG] - Repos.json not modified
[2025-12-28 10:29:42,469 index.py:236 DEBUG] - No config file found in /home/sujith/MLC/repos/local/cache/gate-exam-data, skipping
[2025-12-28 10:29:42,480 index.py:236 DEBUG] - No config file found in /home/sujith/MLC/repos/mlcommons@mlperf-automations/script/reproduce-mlperf-inference-dummy, skipping
[2025-12-28 10:29:42,482 index.py:283 DEBUG] - Deleted keys removed from modified times and indices: set()
[2025-12-28 10:29:42,482 index.py:291 DEBUG] - **************Index unchanged (no changes detected).********************
[2025-12-28 10:29:42,485 script_utils.py:83 INFO] - * mlcr detect,os
[2025-12-28 10:29:42,485 script_utils.py:294 DEBUG] -   - Searching for cached script outputs with tags: -tmp,detect,os
[2025-12-28 10:29:42,485 index.py:32 DEBUG] - Repos path for Index: /home/sujith/MLC/repos
[2025-12-28 10:29:42,490 index.py:174 DEBUG] - index_script.json exists. Skipping forced rebuild.
[2025-12-28 10:29:42,490 index.py:184 DEBUG] - Current repos.json mtime: 1766834865.1588597
[2025-12-28 10:29:42,490 index.py:185 DEBUG] - Old repos.json mtime: 1766834865.1588597
[2025-12-28 10:29:42,490 index.py:204 DEBUG] - Repos.json not modified
[2025-12-28 10:29:42,490 index.py:236 DEBUG] - No config file found in /home/sujith/MLC/repos/local/cache/gate-exam-data, skipping
[2025-12-28 10:29:42,494 index.py:236 DEBUG] - No config file found in /home/sujith/MLC/repos/mlcommons@mlperf-automations/script/reproduce-mlperf-inference-dummy, skipping
[2025-12-28 10:29:42,495 index.py:283 DEBUG] - Deleted keys removed from modified times and indices: set()
[2025-12-28 10:29:42,495 index.py:291 DEBUG] - **************Index unchanged (no changes detected).********************
[2025-12-28 10:29:42,495 module.py:636 DEBUG] -   - Found script::detect-os, 863735b7db8c44fc in /home/sujith/MLC/repos/mlcommons@mlperf-automations/script/detect-os
[2025-12-28 10:29:42,495 module.py:4554 DEBUG] -   - Checking if script execution is already cached ...
[2025-12-28 10:29:42,495 script_utils.py:411 DEBUG] -   - Preparing cache tags...
[2025-12-28 10:29:42,495 script_utils.py:517 DEBUG] -     - Searching for cached script outputs with the following tags: -tmp,detect,os,detect-os,info
[2025-12-28 10:29:42,496 script_utils.py:537 DEBUG] -   - Skipping remembered cache selections...
[2025-12-28 10:29:42,496 script_utils.py:654 DEBUG] -   - Validating cached entry: /home/sujith/MLC/repos/local/cache/detect-os_eebce121
[2025-12-28 10:29:42,496 script_utils.py:573 DEBUG] -   - Validated cached entry: /home/sujith/MLC/repos/local/cache/detect-os_eebce121
[2025-12-28 10:29:42,496 module.py:1027 DEBUG] -     - Found cached script output: /home/sujith/MLC/repos/local/cache/detect-os_eebce121
[2025-12-28 10:29:42,496 module.py:1056 DEBUG] -     - Checking prehook dependencies on other MLC scripts:
[2025-12-28 10:29:42,496 module.py:1069 DEBUG] -       - Loading state from cached entry ...
[2025-12-28 10:29:42,497 module.py:1081 INFO] -      ! load /home/sujith/MLC/repos/local/cache/detect-os_eebce121/mlc-cached-state.json
[2025-12-28 10:29:42,497 module.py:1116 DEBUG] -     - Checking posthook dependencies on other MLC scripts:
[2025-12-28 10:29:42,497 module.py:1129 DEBUG] -     - Checking post dependencies on other MLC scripts:
[2025-12-28 10:29:42,498 module.py:1957 INFO] - {
  "return": 0,
  "env": {
    "MLC_HOST_OS_TYPE": "linux",
    "MLC_HOST_OS_BITS": "64",
    "MLC_HOST_OS_FLAVOR": "pop",
    "MLC_HOST_OS_FLAVOR_LIKE": "ubuntu debian",
    "MLC_HOST_OS_VERSION": "22.04",
    "MLC_HOST_OS_KERNEL_VERSION": "6.17.4-76061704-generic",
    "MLC_HOST_OS_GLIBC_VERSION": "2.35",
    "MLC_HOST_OS_MACHINE": "x86_64",
    "MLC_HOST_OS_PACKAGE_MANAGER": "apt",
    "MLC_HOST_OS_PACKAGE_MANAGER_INSTALL_CMD": "DEBIAN_FRONTEND=noninteractive apt-get install -y",
    "MLC_HOST_OS_PACKAGE_MANAGER_UPDATE_CMD": "apt-get update -y",
    "+MLC_HOST_OS_DEFAULT_LIBRARY_PATH": [
      "/usr/local/lib/x86_64-linux-gnu",
      "/lib/x86_64-linux-gnu",
      "/usr/lib/x86_64-linux-gnu",
      "/usr/lib/x86_64-linux-gnu64",
      "/usr/local/lib64",
      "/lib64",
      "/usr/lib64",
      "/usr/local/lib",
      "/lib",
      "/usr/lib",
      "/usr/x86_64-linux-gnu/lib64",
      "/usr/x86_64-linux-gnu/lib"
    ],
    "MLC_HOST_PLATFORM_FLAVOR": "x86_64",
    "MLC_HOST_PYTHON_BITS": "64",
    "MLC_HOST_SYSTEM_NAME": "ideapad-g3"
  },
  "new_env": {
    "MLC_HOST_OS_TYPE": "linux",
    "MLC_HOST_OS_BITS": "64",
    "MLC_HOST_OS_FLAVOR": "pop",
    "MLC_HOST_OS_FLAVOR_LIKE": "ubuntu debian",
    "MLC_HOST_OS_VERSION": "22.04",
    "MLC_HOST_OS_KERNEL_VERSION": "6.17.4-76061704-generic",
    "MLC_HOST_OS_GLIBC_VERSION": "2.35",
    "MLC_HOST_OS_MACHINE": "x86_64",
    "MLC_HOST_OS_PACKAGE_MANAGER": "apt",
    "MLC_HOST_OS_PACKAGE_MANAGER_INSTALL_CMD": "DEBIAN_FRONTEND=noninteractive apt-get install -y",
    "MLC_HOST_OS_PACKAGE_MANAGER_UPDATE_CMD": "apt-get update -y",
    "+MLC_HOST_OS_DEFAULT_LIBRARY_PATH": [
      "/usr/local/lib/x86_64-linux-gnu",
      "/lib/x86_64-linux-gnu",
      "/usr/lib/x86_64-linux-gnu",
      "/usr/lib/x86_64-linux-gnu64",
      "/usr/local/lib64",
      "/lib64",
      "/usr/lib64",
      "/usr/local/lib",
      "/lib",
      "/usr/lib",
      "/usr/x86_64-linux-gnu/lib64",
      "/usr/x86_64-linux-gnu/lib"
    ],
    "MLC_HOST_PLATFORM_FLAVOR": "x86_64",
    "MLC_HOST_PYTHON_BITS": "64",
    "MLC_HOST_SYSTEM_NAME": "ideapad-g3"
  },
  "state": {
    "os_uname_machine": "x86_64",
    "os_uname_all": "Linux ideapad-g3 6.17.4-76061704-generic #202510191616~1762410050~22.04~898873a SMP PREEMPT_DYNAMIC Thu N x86_64 x86_64 x86_64 GNU/Linux"
  },
  "new_state": {
    "os_uname_machine": "x86_64",
    "os_uname_all": "Linux ideapad-g3 6.17.4-76061704-generic #202510191616~1762410050~22.04~898873a SMP PREEMPT_DYNAMIC Thu N x86_64 x86_64 x86_64 GNU/Linux"
  },
  "deps": []
}
[2025-12-28 10:29:42,498 module.py:1966 DEBUG] -   - running time of script "detect-os,detect,os,info": 0.04 sec.

github-actions · 2025-12-28T05:06:07Z

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅

arjunsuresh · 2025-12-28T22:19:16Z

automation/script/module.py

        if cache:
            # TBD - need to reuse and prune cache_list instead of a new CM
            # search inside find_cached_script
+            ctx = SimpleNamespace(


Why we use a namespace here?

As we need to pass on multiple parameters to find_cached_script function and its sub function call, thatswhy used namespace to obtain a lightweight temporary structure to bundle these arguments into a single ctx object, so that it can be passed on to each sub function calls easily and to avoid using a long dictionary or creating a separate new class just for this purpose.

arjunsuresh · 2025-12-30T01:17:24Z

automation/script/script_utils.py

        c for c in cache_list
        if c.meta.get("associated_script_item_uid") == selected_uid
    ]
+


should we put these functions under a new file cache_utils.py?

I think we can, and we can move _fix_cache_paths there as well

yes, I think everything related to cache can be moved to make the code base better.

Refactor find cached script logic in script automation

ed4798e

sujik18 requested a review from a team as a code owner December 28, 2025 05:05

Update _call_run_deps arguments

8ee496a

arjunsuresh reviewed Dec 28, 2025

View reviewed changes

Revert back to usage of dict from namespace in script automation

2cb5b2b

arjunsuresh reviewed Dec 30, 2025

View reviewed changes

Refactor cache handling through cache_utils.py file

42f12f2

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Refactor find cached script logic in script automation #750

Refactor find cached script logic in script automation #750

sujik18 commented Dec 28, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Dec 28, 2025 •

edited

Loading

Uh oh!

arjunsuresh Dec 28, 2025

Uh oh!

sujik18 Dec 29, 2025

Uh oh!

arjunsuresh Dec 30, 2025

Uh oh!

sujik18 Dec 30, 2025

Uh oh!

arjunsuresh Dec 30, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Refactor find cached script logic in script automation #750

Are you sure you want to change the base?

Refactor find cached script logic in script automation #750

Conversation

sujik18 commented Dec 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🧾 PR Checklist

📁 File Hygiene & Output Handling

📝 Comments & Communication

🛡️ Safety & Security

Uh oh!

github-actions bot commented Dec 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

arjunsuresh Dec 28, 2025

Choose a reason for hiding this comment

Uh oh!

sujik18 Dec 29, 2025

Choose a reason for hiding this comment

Uh oh!

arjunsuresh Dec 30, 2025

Choose a reason for hiding this comment

Uh oh!

sujik18 Dec 30, 2025

Choose a reason for hiding this comment

Uh oh!

arjunsuresh Dec 30, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

sujik18 commented Dec 28, 2025 •

edited

Loading

github-actions bot commented Dec 28, 2025 •

edited

Loading