USHIFT-7041: MicroShift CI Doctor directory restructure

docs/claude/microshift-ci.md

@@ -68,7 +68,7 @@ Alternatively, list all unresolved AI-generated bugs in JIRA using the
 [JIRA query](https://redhat.atlassian.net/issues?jql=project%20%3D%20USHIFT%20AND%20labels%20%3D%20%22microshift-ci-ai-generated%22%20AND%20resolution%20%3D%20Unresolved%20ORDER%20BY%20created%20DESC).
 
 To review what was created, skipped (duplicate, stale regression, infrastructure),
-or already tracked, check the `analyze-ci-create-bugs-merged.txt` file in the
+or already tracked, check the `report-create-bugs.txt` file in the
 working directory.
 
 Each bug should be reviewed and either acted on or closed:

plugins/lvms-ci/skills/doctor/SKILL.md

@@ -47,7 +47,7 @@ Compute once at the start by running `date +%y%m%d` and substituting into the pa
    ```
 
 3. The script deterministically:
-   - For each release: fetches failed periodic jobs, downloads artifacts, writes `<WORKDIR>/analyze-ci-release-<version>-jobs.json`
+   - For each release: fetches failed periodic jobs, downloads artifacts, writes `<WORKDIR>/jobs/release-<version>-jobs.json`
    - Outputs a JSON summary listing all releases, job counts, and file paths
 4. Read the JSON output to know which releases have jobs to analyze and how many
 
@@ -78,7 +78,7 @@ Compute once at the start by running `date +%y%m%d` and substituting into the pa
    Agent: subagent_type=general_purpose, prompt="Analyze this Prow job and save the report:
    1. Run /lvms-ci:prow-job <ARTIFACTS_DIR>
    2. After the analysis completes, save the FULL report output (including the --- STRUCTURED SUMMARY --- block) to:
-      <WORKDIR>/analyze-ci-release-<RELEASE>-job-<N>-<JOB_ID>.txt
+      <WORKDIR>/jobs/release-<RELEASE>-job-<N>-<JOB_ID>.txt
       Use the Write tool to save the file. The file must contain the complete analysis report."
    ```
 
@@ -102,7 +102,7 @@ Compute once at the start by running `date +%y%m%d` and substituting into the pa
 
 2. The script deterministically:
    - Runs `aggregate.py` for each release → `summary.json` files
-   - Runs `create-report.py` → `lvm-operator-ci-doctor-report.html`
+   - Runs `create-report.py` → `report-lvm-operator-ci-doctor.html`
 3. Report the script's output to the user
 
 ### Step 4: Report Completion
@@ -120,7 +120,7 @@ Summary:
     Release main: 3 failed periodic jobs
     Release 4.22: 0 failed periodic jobs
 
-HTML report generated: <WORKDIR>/lvm-operator-ci-doctor-report.html
+HTML report generated: <WORKDIR>/report-lvm-operator-ci-doctor.html
 ```
 
 ## Examples

plugins/microshift-ci/scripts/continue-session.sh

@@ -130,18 +130,15 @@ main() {
     [[ -s "${DL_ERR}" ]] && cat "${DL_ERR}" >&2
 
     # gsutil cp -r creates a subdirectory named "artifacts" inside dl_tmp;
-    # move all files up to the workdir root (flat layout)
+    # copy preserving the source directory structure (jobs/, bugs/)
     local src_dir="${dl_tmp}"
     if [[ -d "${dl_tmp}/artifacts" ]]; then
         src_dir="${dl_tmp}/artifacts"
     fi
 
-    local kept=0
-    for f in "${src_dir}"/*; do
-        [[ -f "${f}" ]] || continue
-        mv "${f}" "${workdir}/"
-        kept=$((kept + 1))
-    done
+    local kept
+    kept=$(find "${src_dir}" -type f | wc -l)
+    cp -a "${src_dir}/." "${workdir}/"
     rm -rf "${dl_tmp}"
 
     if [[ "${kept}" -eq 0 ]]; then
@@ -159,20 +156,20 @@ main() {
 
     # Discover releases from jobs JSON files
     local releases_json="[]"
-    for jobs_file in "${workdir}"/analyze-ci-release-*-jobs.json; do
+    for jobs_file in "${workdir}"/jobs/release-*-jobs.json; do
         [[ -f "${jobs_file}" ]] || continue
         local basename_f
         basename_f=$(basename "${jobs_file}")
-        # Extract release from filename: analyze-ci-release-<VERSION>-jobs.json
+        # Extract release from filename: release-<VERSION>-jobs.json
         local release
-        release=$(echo "${basename_f}" | sed 's/analyze-ci-release-//;s/-jobs\.json//')
+        release=$(echo "${basename_f}" | sed 's/release-//;s/-jobs\.json//')
 
         local job_reports
-        job_reports=$(find "${workdir}" -maxdepth 1 -name "analyze-ci-release-${release}-job-*.txt" | wc -l)
+        job_reports=$(find "${workdir}/jobs" -maxdepth 1 -name "release-${release}-job-*.txt" 2>/dev/null | wc -l)
         local has_summary=false
-        [[ -f "${workdir}/analyze-ci-release-${release}-summary.json" ]] && has_summary=true
+        [[ -f "${workdir}/jobs/release-${release}-summary.json" ]] && has_summary=true
         local has_bugs=false
-        [[ -f "${workdir}/analyze-ci-bugs-${release}.json" ]] && has_bugs=true
+        [[ -f "${workdir}/bugs/bug-matches-${release}.json" ]] && has_bugs=true
 
         releases_json=$(echo "${releases_json}" | jq \
             --arg r "${release}" \
@@ -184,11 +181,11 @@ main() {
 
     # PR info
     local prs_json="null"
-    if [[ -f "${workdir}/analyze-ci-prs-jobs.json" ]]; then
+    if [[ -f "${workdir}/jobs/prs-jobs.json" ]]; then
         local pr_reports
-        pr_reports=$(find "${workdir}" -maxdepth 1 -name "analyze-ci-prs-job-*.txt" | wc -l)
+        pr_reports=$(find "${workdir}/jobs" -maxdepth 1 -name "prs-job-*.txt" 2>/dev/null | wc -l)
         local pr_has_summary=false
-        [[ -f "${workdir}/analyze-ci-prs-summary.json" ]] && pr_has_summary=true
+        [[ -f "${workdir}/jobs/prs-summary.json" ]] && pr_has_summary=true
         prs_json=$(jq -n \
             --argjson jr "${pr_reports}" \
             --argjson hs "${pr_has_summary}" \
@@ -197,8 +194,8 @@ main() {
 
     # HTML report path
     local html_report="null"
-    if [[ -f "${workdir}/microshift-ci-doctor-report.html" ]]; then
-        html_report="\"${workdir}/microshift-ci-doctor-report.html\""
+    if [[ -f "${workdir}/report-microshift-ci-doctor.html" ]]; then
+        html_report="\"${workdir}/report-microshift-ci-doctor.html\""
     fi
 
     # Final JSON

plugins/microshift-ci/scripts/search-bugs.py

@@ -16,7 +16,7 @@
       - PR number: pr-6396, pr6396
       - Rebase shorthand: rebase-release-4.22
 
-    --merge mode reads multiple analyze-ci-bug-candidates-<source>.json
+    --merge mode reads multiple bug-candidates-<source>.json
     files and merges candidates across sources using fuzzy signature
     matching for cross-release dedup.
 
@@ -25,9 +25,10 @@
     text report.
 
 Output:
-    ${WORKDIR}/analyze-ci-bug-candidates-<source>.json       (default mode)
+    ${WORKDIR}/bugs/bug-candidates-<source>.json  (default mode)
     <output>                                                   (--merge mode, via --output)
-    ${WORKDIR}/analyze-ci-create-bugs-{<source>|merged}.txt  (--report mode)
+    ${WORKDIR}/bugs/create-bugs-<source>.txt   (--report mode, per-source)
+    ${WORKDIR}/report-create-bugs.txt                       (--report mode, merged)
 """
 
 import json
@@ -299,18 +300,21 @@ def find_job_files(workdir, source):
     """Find per-job report files for a given source.
 
     Returns (files, source_label) tuple.
+    Job reports live under ${workdir}/jobs/.
     """
+    jobs_dir = os.path.join(workdir, "jobs")
+
     # Release version
     if re.match(r"^(\d+\.\d+|main)$", source):
-        pattern = os.path.join(workdir, f"analyze-ci-release-{source}-job-*.txt")
+        pattern = os.path.join(jobs_dir, f"release-{source}-job-*.txt")
         files = sorted(glob_mod.glob(pattern))
         return files, f"release {source}"
 
     # PR number
     m = re.match(r"^pr-?(\d+)$", source)
     if m:
         pr_num = m.group(1)
-        pattern = os.path.join(workdir, f"analyze-ci-prs-job-*-pr{pr_num}-*.txt")
+        pattern = os.path.join(jobs_dir, f"prs-job-*-pr{pr_num}-*.txt")
         files = sorted(glob_mod.glob(pattern))
         return files, f"PR #{pr_num}"
 
@@ -322,7 +326,7 @@ def find_job_files(workdir, source):
 
         # Find PR numbers for this rebase source from the status file
         rebase_pr_numbers = set()
-        status_file = os.path.join(workdir, "analyze-ci-prs-status.json")
+        status_file = os.path.join(jobs_dir, "prs-status.json")
         if os.path.isfile(status_file):
             with open(status_file, "r") as f:
                 try:
@@ -335,7 +339,7 @@ def find_job_files(workdir, source):
                     if pr_num is not None:
                         rebase_pr_numbers.add(int(pr_num))
 
-        pattern = os.path.join(workdir, "analyze-ci-prs-job-*.txt")
+        pattern = os.path.join(jobs_dir, "prs-job-*.txt")
         all_files = sorted(glob_mod.glob(pattern))
         files = []
         for filepath in all_files:
@@ -425,12 +429,13 @@ def union(a, b):
 
 
 def _load_jira_lookup(workdir):
-    """Load Jira duplicates/regressions from bug mapping files in workdir.
+    """Load Jira duplicates/regressions from bug mapping files.
 
     Returns a dict mapping error_signature to {duplicates, regressions}.
+    Bug mapping files live under ${workdir}/bugs/.
     """
     lookup = {}
-    pattern = os.path.join(workdir, "analyze-ci-bugs-*.json")
+    pattern = os.path.join(workdir, "bugs", "bug-matches-*.json")
     for filepath in sorted(glob_mod.glob(pattern)):
         with open(filepath, "r") as f:
             data = json.load(f)
@@ -460,7 +465,7 @@ def merge_candidate_files(filepaths, workdir=None):
     analysis_text) and post-Jira bug mapping files (duplicates, regressions).
 
     When workdir is provided and contains bug mapping files
-    (analyze-ci-bugs-*.json), their Jira data is injected into candidates
+    (bug-matches-*.json), their Jira data is injected into candidates
     so that _merge_groups_by_jira() can merge groups sharing issue keys.
 
     Returns a dict with sources, total_candidates, and candidates[] where
@@ -846,9 +851,14 @@ def main_report(report_file, candidates_file, workdir):
         tag = release_sources[0]
     else:
         tag = "merged" if len(sources) > 1 else sources[0]
-    filename = f"analyze-ci-create-bugs-{tag}.txt"
-
-    output_path = os.path.join(workdir, filename)
+    if tag == "merged":
+        filename = "report-create-bugs.txt"
+        output_path = os.path.join(workdir, filename)
+    else:
+        filename = f"create-bugs-{tag}.txt"
+        bugs_dir = os.path.join(workdir, "bugs")
+        os.makedirs(bugs_dir, exist_ok=True)
+        output_path = os.path.join(bugs_dir, filename)
     report_with_footer = report + f"\n\nReport saved: {output_path}\n{SEPARATOR}\n"
 
     with open(output_path, "w") as f:
@@ -982,7 +992,9 @@ def main():
         "candidates": candidates,
     }
 
-    output_path = os.path.join(workdir, f"analyze-ci-bug-candidates-{source}.json")
+    bugs_dir = os.path.join(workdir, "bugs")
+    os.makedirs(bugs_dir, exist_ok=True)
+    output_path = os.path.join(bugs_dir, f"bug-candidates-{source}.json")
     with open(output_path, "w") as f:
         json.dump(result, f, indent=2)
 
@@ -1013,6 +1025,8 @@ def main_merge(merge_files, output_file, workdir):
         sys.exit(1)
 
     os.makedirs(workdir, exist_ok=True)
+    bugs_dir = os.path.join(workdir, "bugs")
+    os.makedirs(bugs_dir, exist_ok=True)
 
     print(f"Merging {len(merge_files)} candidate files", file=sys.stderr)
     result = merge_candidate_files(merge_files, workdir=workdir)

plugins/microshift-ci/skills/close-stale-bugs/SKILL.md

@@ -38,7 +38,7 @@ Operates in **dry-run mode by default** — shows which bugs would be closed wit
 ## Prerequisites
 
 - An existing workdir from a prior `/microshift-ci:doctor` run (today's date)
-- `analyze-ci-bugs-summary.json` must exist in the workdir (produced by the doctor finalize step)
+- `bugs/bug-matches-summary.json` must exist in the workdir (produced by the doctor finalize step)
 - MCP Jira server must be configured and accessible (for `--close` mode)
 
 ## Work Directory
@@ -55,10 +55,10 @@ Compute once at the start by running `date +%y%m%d` and substituting into the pa
 
 1. Parse `<ARGUMENTS>` for the `--close` flag. If present, set MODE to `close`; otherwise MODE is `dry-run`.
 2. Compute today's `<WORKDIR>` by running `date +%y%m%d` and substituting into `/tmp/microshift-ci-claude-workdir.<YYMMDD>`.
-3. Read `<WORKDIR>/analyze-ci-bugs-summary.json`. If the file does not exist, report a **fatal error** and stop:
+3. Read `<WORKDIR>/bugs/bug-matches-summary.json`. If the file does not exist, report a **fatal error** and stop:
 
    ```text
-   Error: analyze-ci-bugs-summary.json not found in <WORKDIR>
+   Error: bugs/bug-matches-summary.json not found in <WORKDIR>
    Run the full doctor workflow first: /microshift-ci:doctor <releases>
    ```
 
@@ -192,7 +192,7 @@ Actually closes all matching bugs in JIRA.
 
 ## Notes
 
-- This skill does NOT re-analyze jobs or re-query JIRA for bug lists — it reads the pre-computed `analyze-ci-bugs-summary.json` from the doctor finalize step
+- This skill does NOT re-analyze jobs or re-query JIRA for bug lists — it reads the pre-computed `bugs/bug-matches-summary.json` from the doctor finalize step
 - The `unlinked[]` array in the summary file contains bugs that are open, AI-generated, and not matched to any current CI failure signature
 - Bugs with an assignee are never closed — someone has picked up the work
 - The 10-day staleness threshold ensures recently-created or recently-commented bugs are not prematurely closed

plugins/microshift-ci/skills/continue-session/SKILL.md

@@ -16,7 +16,7 @@ allowed-tools: Bash, Read, Glob, Grep
 
 ## Description
 
-Downloads CI Doctor analysis artifacts (per-job reports, summaries, bug mappings, HTML report) from a completed prow job into a local workdir. The workdir date is derived from the prow job's start timestamp, matching the layout the doctor skill creates. This lets you pick up where the CI agent left off — inspect reports, re-run aggregation, create bugs, or do further investigation.
+Downloads CI Doctor analysis artifacts (per-job reports, summaries, bug mappings, HTML report) from a completed prow job into a local workdir, preserving the source directory structure. The workdir date is derived from the prow job's start timestamp, matching the layout the doctor skill creates. This lets you pick up where the CI agent left off — inspect reports, re-run aggregation, create bugs, or do further investigation.
 
 ## Arguments
 
@@ -34,7 +34,10 @@ Downloads CI Doctor analysis artifacts (per-job reports, summaries, bug mappings
    - Parses the URL and converts it to a GCS path
    - Fetches `started.json` to derive the job date → workdir path
    - Fails if the workdir already exists (prevents clobbering local data)
-   - Downloads analysis files (`analyze-ci-*`, HTML report, claude logs)
+   - Downloads analysis files into subdirectories preserving the source structure:
+     - `<WORKDIR>/jobs/` — job analysis files (`release-*`, `prs-*`)
+     - `<WORKDIR>/bugs/` — bug correlation files (`bugs-*`, `bug-candidates-*`)
+     - `<WORKDIR>/` — final reports (HTML report, claude logs)
    - Outputs a JSON summary to stdout
 
 2. Read the JSON summary. It contains:
@@ -51,7 +54,7 @@ Downloads CI Doctor analysis artifacts (per-job reports, summaries, bug mappings
 
 4. Suggest next actions based on what's available:
 
-   - **View the HTML report**: `open <workdir>/microshift-ci-doctor-report.html`
+   - **View the HTML report**: `open <workdir>/report-microshift-ci-doctor.html`
    - **Re-generate the HTML report** (e.g., after modifying job reports):
 
      ```text
@@ -69,7 +72,7 @@ Downloads CI Doctor analysis artifacts (per-job reports, summaries, bug mappings
    - **Read individual job reports** for deeper investigation:
 
      ```text
-     <WORKDIR>/analyze-ci-release-<VERSION>-job-<N>-<BUILD_ID>.txt
+     <WORKDIR>/jobs/release-<VERSION>-job-<N>-<BUILD_ID>.txt
      ```
 
    - **Re-analyze a specific prow job** (downloads fresh artifacts):
@@ -86,5 +89,5 @@ Downloads CI Doctor analysis artifacts (per-job reports, summaries, bug mappings
 ## Notes
 
 - Only analysis files are downloaded — raw prow job artifacts (build logs, SOS reports) are not included. Use `/microshift-ci:prow-job` or `download-jobs.sh` to fetch those for specific jobs.
-- The workdir uses the same layout as the doctor skill (`/tmp/microshift-ci-claude-workdir.<YYMMDD>`), so all doctor scripts (`finalize`, `aggregate.py`, `create-report.py`) work directly on the downloaded data.
+- The workdir uses the same layout as the doctor skill (`/tmp/microshift-ci-claude-workdir.<YYMMDD>`), with job analysis files under `jobs/` and bug correlation files under `bugs/`. All doctor scripts (`finalize`, `aggregate.py`, `create-report.py`) work directly on the downloaded data.
 - If the workdir for the job's date already exists, the script exits with an error. Remove the existing workdir first if you want to replace it with CI data.