Skip to content

bug(test): test_development_speedup swallows 3 of its 4 sub-test failures and still reports success #14931

Closed

Description

@mrveiss

test_development_speedup reports success when 3 of its 4 sub-tests fail

Found reviewing PR #14929. That PR converted the last of four swallowing handlers in this function into a raise; the other three still log and continue.

Duplicated verbatim in two files:

  • autobot-infrastructure/shared/scripts/analysis/test_npu_code_search.py (~lines 144, 164, 184 still swallowing; 212 converted)
  • autobot-npu-worker/npu_code_search_test.py (~lines 142, 162, 182 still swallowing; 210 converted)

The shape

try:
    duplicates = find_duplicate_code(...)      # Test 1
except Exception as e:
    logger.error(f"   Duplicate detection failed: {e}")   # swallowed, continues

Four such blocks: duplicate detection, pattern analysis, import analysis, comprehensive analysis. Only the fourth now raises. So if find_duplicate_code(), identify_code_patterns() or analyze_imports_and_dependencies() throws, the function still reaches its return True and the driver counts the test as passed.

That means the assertion #14929 added is narrower than it looks: it catches failure of the last sub-test only. Three quarters of what this test exercises can fail silently.

Why it was scoped out of #14929

#14929's remit was the driver-consumption defect — nine tests whose return True was converted to a bare return, breaking the driver's verdict. Un-swallowing the other three handlers is a behaviour change to what those tests assert, which is a different question and deserves its own review. Recorded here rather than folded in.

Note on duplication

These two files are near-identical copies. Whatever fix lands must land in both, or they drift — that is the [union-merge] concern that produced #14873 and #14886. Consider whether one should import the other rather than being maintained twice.

Acceptance

  • A failure in any of the four sub-tests fails the test, not just the fourth
  • The driver still distinguishes pass from fail (it sums truthiness over result = await test_func(), so the function must keep returning a truthy value on success — see bug(test): 262 tests return a value instead of asserting, so they cannot fail #14920)
  • Both copies fixed, or the duplication resolved so there is one copy
  • Mutation-proved: break each of the four sub-calls in turn and confirm the test goes red for each

Related: #14920 (returns instead of asserting), #14930 (the marker suite that cannot collect), #14873 / #14886 (duplicate modules that drift).

Activity

  1. github-actions commented on Aug 24, 2026

    @github-actions
    Contributor

    PR #14980 (merged to Dev_new_gui) references this issue with a close keyword.

    fix(test): three checks that could not report what they were written to report (#14931, #14941, #14927)

    If this issue is fully resolved, close it manually. If work remains, no action is needed.

  2. mrveiss commented on Aug 24, 2026

    @mrveiss
    OwnerAuthor

    Delivered and verified in Dev_new_gui.

    PR #14980 (squashed)
    Merge commit 0256c96dd6e3891e85964494dc83f2447d443528
    In base confirmed via git merge-base --is-ancestor
    CI at merge 10/10 required contexts green, 47 success, 0 failures, 0 behind base

    What landed

    All four except Exception blocks in test_development_speedup now
    raise AssertionError(...) from e instead of swallowing the failure, applied
    identically in both near-identical copies
    (autobot-infrastructure/shared/scripts/analysis/test_npu_code_search.py and
    autobot-npu-worker/npu_code_search_test.py). Verified present in base: 4
    raise AssertionError sites.

    The acceptance criterion "driver still distinguishes pass from fail" is preserved:
    main() sums truthiness (result = await test_func(); if result: success_count += 1),
    so the function still ends return True. The return contract was kept deliberately
    rather than removed, because that driver reads it — unlike #14941's, which does not.

    Closing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingtesting

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions