scripts: add rule-prevalence.py to detect FP-prone rules

[kami922]

closes #424

Added scripts/rule-prevalence.py ,a small utility that runs capa rules against a folder of .frz freeze files and reports how often each rule fires. The idea is to help rule reviewers spot rules that match too broadly across clean reference samples (a signal for potential false positives). Rules hitting >= 50% of samples are highlighted in red. Supports --quiet to hide zero-hit rules and --threshold to tune the cutoff.

Usage:

# generate a freeze file from a binary
python -c "import capa.features.freeze; capa.features.freeze.main(['sample.exe', 'sample.exe.frz'])"

# run the script
python scripts/rule-prevalence.py /path/to/frz/dir/ --rules rules/ -q

Tests:
pytests passes locally.

Checklist

• No CHANGELOG update needed
• No new tests needed
• No documentation update needed
• This submission includes AI-generated code and I have provided details in the description.

[gemini-code-assist]

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces a valuable new tool for capa rule developers and reviewers. The rule-prevalence.py script provides an automated way to assess the behavior of capa rules against a collection of reference samples. By quantifying how frequently rules match, it facilitates the identification of overly broad or 'dead' rules, thereby improving the overall quality and precision of the rule set without requiring manual inspection of each rule's output.

Highlights

• New Utility Script: A new Python script, scripts/rule-prevalence.py, has been added to help identify false-positive prone capa rules.
• Rule Prevalence Analysis: The script runs capa rules against a directory of .frz freeze files and reports how often each rule fires, indicating its prevalence across samples.
• False Positive Detection: Rules matching 50% or more of the analyzed samples are highlighted in red, signaling potential false positives that may match too broadly.
• Configurable Options: The script supports --quiet to hide rules that do not match any files and --threshold (though not explicitly implemented in the provided diff, it's mentioned in the description's intent) to tune the cutoff for highlighting.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Changelog

• CHANGELOG.md
  • Added an entry for the new rule-prevalence.py script under 'New Features'.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[google-cla]

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

[gemini-code-assist]

Code Review

This pull request introduces a new script, rule-prevalence.py, to analyze the frequency of capa rule matches across a set of freeze files. This is a useful utility for identifying potentially false-positive-prone rules. The script is well-structured and uses rich for clear output.

My review includes two main points for improvement:

1. Implementing the --threshold command-line option, which is mentioned in the pull request description but is currently missing.
2. A suggestion to use pathlib.Path directly in argparse for cleaner path handling.

Overall, this is a valuable addition to the project's tooling.

[gemini-code-assist]

The PR description mentions a configurable --threshold, but a hardcoded value of 50 is used here. To implement this feature, you should:

1. Add a --threshold argument in main():

parser.add_argument("--threshold", type=int, default=50, help="threshold percentage to highlight rules (default: 50)")

2. Update render_table's signature to accept the threshold:

def render_table(counts: dict[str, int], rules: capa.rules.RuleSet, total: int, quiet: bool, threshold: int) -> None:

3. Pass the threshold from main() to render_table():

render_table(counts, rules, total=len(frz_paths), quiet=args.quiet, threshold=args.threshold)

4. Use the threshold parameter here.

Suggested change

	row_style = "red" if rate >= 50 else ""
	row_style = "red" if rate >= threshold else ""

[gemini-code-assist]

Since this project uses Python 3.10+, you can use pathlib.Path directly as a type for argparse arguments. This simplifies path handling, as you would no longer need to explicitly convert the string arguments to Path objects later in the code (e.g., Path(args.input) on line 159 and Path(args.rules) on line 173 would become args.input and args.rules respectively).

Suggested change

	parser.add_argument("input", type=str, help="path to directory containing .frz files")
	parser.add_argument(
	"-r", "--rules", type=str, default=None, help="path to rules directory (uses ./rules if not set)"
	)
	parser.add_argument("input", type=Path, help="path to directory containing .frz files")
	parser.add_argument(
	"-r", "--rules", type=Path, default=None, help="path to rules directory (uses ./rules if not set)"
	)

[kami922]

@mr-tz , @mike-hunhoff @williballenthin Hello I would like to ask question here.
Gemini suggested 2 improvements I would like to know which one would be a better approach from below.

1. Make the changes locally and push into single commit.
2. Simply click on Commit suggestion button 2 times which results in 2 separate commits implementing the suggestion.

[williballenthin]

since the recommendations are for two different things, using separate commits from the GitHub UI (with appropriate commit messages) is probably a better approach.

[williballenthin]

i'd propose that we close this PR for now. writing the script to identify the prevalence is not the blocker to #424. rather, its the collection and maintenance of a representative sample set that we evaluate in CI.