Skip to content

[FEATURE] AI/LLM application testing skills (new vulnerability class) #691

Description

@teedonk

Is your feature request related to a problem? Please describe.
Strix covers web and infra vulnerabilities well but has no skills for the AI/LLM
application attack surface, where untrusted external inputs manipulate a model
through the content it ingests. Based on research, I wrote a piece on this attack surface a few
months ago (https://hackernoon.com/ai-apps-have-a-new-attack-surface-external-inputs),
and the same classes are now shipping in production apps that Strix cannot
currently reach: targets increasingly ship RAG pipelines and tool-calling agents
with none of it covered. report/sarif.py already fingerprints prompt injection
in _VULN_CLASS_KEYWORDS, but no skill exists to actually detect or validate it,
and no other AI class is covered.

Describe the solution you'd like
New skills covering the AI/LLM vulnerability points:

  • Prompt injection, direct and indirect, including affirmative-prefix jailbreaks
  • RAG / knowledge-base poisoning (planted canary-document retrieval)
  • Excessive agency and insecure tool/action handling (missing approval on high-risk agent actions)
  • Document and visual prompt injection (hidden text in PDFs and images)
  • Sensitive information disclosure via the model (PII leakage, model-inversion probing)

Plus a recon skill to fingerprint LLM / RAG / agent / MCP stacks so scans route
into the above. Insertion points: new files under skills/vulnerabilities/ (or a
new skills/ai/ category), any dedicated tooling wired into _BASE_TOOLS in
agents/factory.py, and _VULN_CLASS_KEYWORDS additions in report/sarif.py
for the classes not already listed.

Describe alternatives you've considered

  • Folding these into existing skills (e.g. injection): rejected, as the discovery
    and validation methods differ enough to warrant their own skills.
  • A dedicated skills/ai/ category vs skills/vulnerabilities/: open question,
    happy to follow maintainer preference.

Additional context

Open questions for alignment:

  1. skills/vulnerabilities/ or a dedicated skills/ai/ category?
  2. Stack fingerprinting as a standalone tool, or just a skill?
  3. Preferred naming for the new AI classes in _VULN_CLASS_KEYWORDS?

Happy to implement. If there is appetite I will open a first PR with a single
prompt_injection.md skill in the existing vuln-skill format as a proof of
shape, then land the rest one skill per PR against the checklist above. Keen to
align on the questions before I start.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions