Skip to content

Fix test_all_artifacts_have_required_fields silently skipping half th… - #1

Merged
scriptedstatement merged 2 commits into
AppliedIR:mainfrom
mpilking:fix-artifact-schema-test
Oct 1, 2026
Merged

scriptedstatement merged 2 commits into
AppliedIR:mainfrom
mpilking:fix-artifact-schema-test

Conversation

@mpilking

@mpilking mpilking commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

Problem

test_all_artifacts_have_required_fields validates 27 of 53 artifacts. The other 26 are silently skipped.

The test derives a filename from each artifact's name field, then gives up when the lookup misses:

art = loader.get_artifact(name.lower().replace(" ", "_").replace("-", "_"))
if art is None:
    # Try the description-based name
    continue

That only resolves when an artifact's name directly derives to its filename, which often fails:

┌────────────────────┬────────────────────┬───────────────────┐
│        name        │      derives       │  actual filename  │
├────────────────────┼────────────────────┼───────────────────┤
│ Authentication Log │ authentication_log │ auth_log.yaml     │
├────────────────────┼────────────────────┼───────────────────┤
│ BASH History       │ bash_history       │ bash_history.yaml │
├────────────────────┼────────────────────┼───────────────────┤
│ Cron Jobs          │ cron_jobs          │ crontab.yaml      │
├────────────────────┼────────────────────┼───────────────────┤

Every miss hits the continue and the artifact goes unchecked with no signal. 22 Windows and 4 Linux artifacts are currently affected.

Fix

Read the artifact directories directly, matching the seven other _load_all_in_dir loops already in this file — this was the only artifact test not using that idiom.

Result

Coverage goes from 27 to all 53 artifacts, and the test passes unchanged: every artifact does carry name, description and platform. No artifact files are modified.

🤖 Generated with Claude Code (https://claude.com/claude-code)

…e artifacts

The test derived a filename from each artifact's name field and skipped the
artifact when the lookup missed:

    art = loader.get_artifact(name.lower().replace(" ", "_").replace("-", "_"))
    if art is None:
        continue

That only resolves when an artifact's name round-trips to its filename, which
is not a property the data has or should need. 'Authentication Log' derives
authentication_log but the file is auth_log.yaml; 'Cron Jobs' derives cron_jobs
against crontab.yaml; 'Login Records' derives login_records against
wtmp_utmp.yaml. Each miss hit the continue and the artifact went unchecked
without any signal.

The result was that a test named test_all_artifacts_have_required_fields
validated 27 of 53 artifacts — 22 Windows and 4 Linux were skipped.

Read the artifact directories directly instead, matching the seven other
_load_all_in_dir loops already in this file. Coverage goes to all 53 and the
test passes unchanged, since every artifact does carry name, description and
platform. This is the only artifact test in the file that was not already
using that idiom.

The platform tuple includes "macos" to match loader.py, which enumerates
("windows", "linux", "macos") in both list_artifacts and get_artifacts_for_tool.
_load_all_in_dir returns an empty list for a directory that does not exist, so
this is inert today and the test picks up macOS artifacts automatically if that
directory is ever added — rather than silently skipping them, which is the bug
this commit fixes.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@scriptedstatement
scriptedstatement merged commit 5184811 into AppliedIR:main Oct 1, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants