Skip to content

ci(phase-validation): 'UI/UX 100%' measures only that App.vue and a components dir exist — CI skips every real check, and feature checks point at the deleted autobot-vue/ tree #17089

Closed
#17505
Closed
ci(phase-validation): 'UI/UX 100%' measures only that App.vue and a components dir exist — CI skips every real check, and feature checks point at the deleted autobot-vue/ tree#17089
#17505
Labels
bugSomething isn't workinginfrastructurepriority: mediumtech-debt
Milestone
v0.9.0

Description

@mrveiss
mrveiss
opened on Sep 19, 2026

Problem

The Phase Validation workflow (.github/workflows/phase_validation.yml → autobot-infrastructure/shared/scripts/phase_validation_system.py) reports a completion percentage per phase. "Phase 6: Enhanced UI/UX" shows 100% while the owner observes that the UI is not consistent. The number measures something else.

  • In CI it only checks that files exist. ci_mode (discovery(infra): phase_validation_system.py criteria reference legacy paths #7496) skips every endpoint, service and feature check, because CI has no live stack. So the UI/UX 100% means autobot-frontend/src/App.vue and package.json exist and autobot-frontend/src/components/ is a directory. It says nothing about function, consistency, i18n, theming or UX.

  • The feature checks point at paths that no longer exist, so even outside CI they can't measure the real UI (phase_validation_system.py:757-759):

    • chat_interface → (root / "autobot-vue/src/components").exists()
    • terminal_interface → glob("autobot-vue/src/components/*Terminal*")
    • settings_panel → glob("autobot-vue/src/components/*Settings*")

    autobot-vue/ was renamed to autobot-frontend/. Other checks also use stale paths: src/agents, scripts/monitoring_system.py and scripts/performance_dashboard.py.

  • A bare "100% complete" is a did-not-look result presented as a nothing-missing result (MEASUREMENT_DISCIPLINE).

The workflows that actually test the UI are frontend-test.yml (unit tests), visual-regression.yml, stylelint-tokens.yml (design tokens) and frontend-typecheck-regression.yml. Phase validation doesn't consult any of them.

Acceptance criteria

  • In CI mode the report no longer shows a "completion %". It labels the score structural presence and lists every skipped check as "not checked (needs live stack)". No phase can read 100% while checks were skipped.
  • Every feature check points at paths that exist on main, or is removed. Add a guard test that fails if any path referenced in the phase criteria is missing, so the paths can't go stale again.
  • The UI/UX phase reports the verdicts of the real UI gates (frontend unit tests, visual regression, token lint, typecheck) instead of file presence. Or the phase links to them and doesn't report its own percentage.
  • A test proves that a phase with skipped checks can't be reported as complete.

Activity

  1. mrveiss
    added this to the v0.9.0 milestone on Sep 19, 2026
  2. mrveiss
    added
    bugSomething isn't working
    priority: medium
    tech-debt
    on Sep 19, 2026
  3. mrveiss
    added a parent issue on Sep 19, 2026
  4. github-actions
    added
    infrastructure
    on Sep 19, 2026
  5. mrveiss
    added 3 commits that reference this issue on Sep 25, 2026

    fix(guards): my path guard passed while four checks were dead (#17089)

    6b88a39

    fix(dashboard,tracker): migrate the report's consumers to the honest …

    4b2ef19

    fix(guards): stop leaking phase_score into sys.modules (#17089)

    fe1ceb7
  6. mrveiss
    closed this as completedin #17505on Sep 26, 2026
  7. mrveiss
    added a commit that references this issue on Sep 27, 2026

    fix: restore 32 files my own `git add -A` deleted (#17571)

    b8bcf57

  8. mrveiss commented on Sep 27, 2026

    @mrveiss
    mrveiss
    on Sep 27, 2026
    OwnerAuthor

    Acceptance evidence, verified against merged code — 4 of 4 met

    Closed by #17505. Neither the issue nor the PR body carried a criterion-by-criterion argument; this is it, against origin/main blobs.

    • In CI mode the report no longer shows a "completion %". It labels the score structural presence and lists every skipped check as "not checked (needs live stack)".
      autobot-infrastructure/shared/scripts/phase_score.py:45 defines NOT_CHECKED = "not checked (needs live stack)"; :157 emits structural_presence_percentage; :195-196 emits structural_presence always and sets "measures": "completion" if not skipped_groups else "structural presence only"; :177 states the rule — overall_maturity is emitted only when nothing was skipped. Pinned by repo_tests/phase_validation_paths_and_skips_17089_test.py:248 (test_the_report_omits_completion_percentage_entirely_when_anything_was_skipped), :258 (a full run still reports completion, so the change is conditional rather than a blanket removal) and :282 (overall_maturity is None when a group was skipped).

    • Every feature check points at paths that exist on main, or is removed, with a guard test.
      :211 test_every_criteria_path_exists and :224 test_every_feature_check_path_exists, each preceded by a parse-found control — :203 and :216 — so an extractor that silently found nothing fails instead of passing an empty sweep.

    • The UI/UX phase reports the verdicts of the real UI gates instead of file presence — or links to them and doesn't report its own percentage.
      The second branch was taken, deliberately and in writing. phase_validation_system.py:217-221 states it: "the checks below can only see that files exist. These workflows are what actually exercise this phase, and the report links to them instead of this phase claiming a UI verdict of its own. A file-presence sweep has no business reporting on UX." authoritative_gates (:222-226) names frontend-test.yml, visual-regression.yml, stylelint-tokens.yml, frontend-typecheck-regression.yml — all four exist in origin/main, checked individually. The phase keeps its files/directories keys, which is what a reader stopping there would call a failure; the percentage suppression from the first criterion is what makes the link the reported verdict rather than a footnote.

    • A test proves that a phase with skipped checks can't be reported as complete.
      :239 test_a_phase_with_a_skipped_group_is_not_complete_even_at_100_percent — the explicit case where the ratio of what ran is 100% and the phase still is not complete. Supported by :269 (the ratio is of what ran) and :272 (nothing ran is zero and not complete), which is the did not look versus found nothing distinction in its own right.

    What this closure does not cover

    • The authoritative_gates paths are parsed by the guard (:154, :165) but I did not find an assertion that each one exists, the way :211/:224 assert for criteria and feature-check paths. All four exist today — verified here — so nothing is broken; a workflow renamed later would leave a dead link in the report with no test objecting.
    • The report links to the gates. It does not read their verdicts, so a red visual-regression run does not change what this phase says about itself; it only points a reader at where the real answer lives.
  9. mrveiss
    added a commit that references this issue on Sep 27, 2026

    feat(canonical): the audit harness reported clean over a tree it neve…

    969c195

  10. mrveiss commented on Oct 2, 2026

    @mrveiss
    mrveiss
    on Oct 2, 2026
    OwnerAuthor

    AC verification — 4 of 4 delivered, now ticked

    Verified against git show origin/main: at 2b88c1c2e5.

    AC1 — no completion %, labelled structural presence, skips named. autobot-infrastructure/shared/scripts/phase_score.py:45 defines NOT_CHECKED = "not checked (needs live stack)" — the criterion's wording exactly — and :193-198:

    skipped_groups = sorted({group for score, _ in counted for group in score.skipped})
"measures": "completion" if not skipped_groups else "structural presence only",
"checks_skipped": len(skipped_groups),
"skipped_detail": {group: NOT_CHECKED for group in skipped_groups},

    So the moment anything is skipped the figure stops calling itself completion and every skipped group is listed by name. phase_validation_system.py:411 emits the "%.1f%% structural presence" form.

    AC2 — referenced paths must exist, with a guard. repo_tests/phase_validation_paths_and_skips_17089_test.py:200 class TestEveryReferencedPathExists — :211 test_every_criteria_path_exists and :224 test_every_feature_check_path_exists. Crucially it also carries :203 test_the_parse_found_the_criteria_paths and :216 test_the_parse_found_the_lambda_paths: a reach floor on its own parser, so a parse that silently found nothing cannot pass as "all paths exist".

    AC3 — the second disjunct, chosen explicitly. The criterion allowed either reporting the real gates' verdicts, or "the phase links to them and doesn't report its own percentage." phase_validation_system.py:218-226 takes the second and says so, labelled with the criterion number:

    # #17089 AC3: the checks below can only see that files exist. These
# workflows are what actually exercise this phase, and the report
# links to them instead of this phase claiming a UI verdict of its
# own. A file-presence sweep has no business reporting on UX.
"authoritative_gates": [
    ".github/workflows/frontend-test.yml",
    ".github/workflows/visual-regression.yml",
    ".github/workflows/stylelint-tokens.yml",
    ".github/workflows/frontend-typecheck-regression.yml",
],

    All four gates the criterion names are present — frontend unit tests, visual regression, token lint, typecheck — and AC1's mechanism is what stops the phase reporting a percentage of its own.

    AC4 — skipped can never read as complete. :236 class TestSkippedCanNeverReadAsComplete: :239 test_a_phase_with_a_skipped_group_is_not_complete_even_at_100_percent asserting score.complete is False, :248 the report omits the completion percentage entirely when anything was skipped, :272 test_nothing_ran_is_zero_and_not_complete, and :258 test_a_full_run_still_reports_completion as the negative control, so the check is not simply refusing everything.

    Beyond the criteria, two things worth recording because they are the same discipline one level out: :279 class TestTheOverallFiguresDistinguishTheTwo asserts overall maturity is None when a group was skipped and that ready_for_production is None rather than False when unassessed — unknown kept distinct from negative. And :316 class TestTheSkipListStaysInStepWithWhatIsSkipped keeps the skip list from going stale.

    A note on how this was judged. AC3 is disjunctive, and a verifier should decline rather than pick the reading that resolves. It did not need declining: the code names the criterion and states which branch it took, so the answer was reachable from merged code. Declining when the answer is reachable would be its own failure.

    Part of a sweep over the 93 v0.9.0 closures closed COMPLETED with a checklist and nothing ticked (#17361). Drawn in a seeded random sample.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workinginfrastructurepriority: mediumtech-debt

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions