Detect wildcards during reporting

[kazet]

so that if the same problem exists on a.example.com and b.example.com and anydomain.example.com, only one instance is reported

[Nazihbenbrahim]

Hi @kazet, I'd love to work on this issue! Is it available for me to take?

[Nazihbenbrahim]

Implementation Steps

1. Added get_root_domain Method

   This method extracts the root domain (wildcard) from a subdomain or URL.

   Example:

    Input: a.example.com → Output: *.example.com
   
    Input: example.com → ### **Output:** example.com
   

def get_root_domain(self) -> Optional[str]:
"""
Extracts the root domain (wildcard) from the target.
For example, a.example.com -> *.example.com.
"""
if self.target_is_url():
hostname = urllib.parse.urlparse(self.target).hostname
else:
hostname = self.target

if not hostname:
    return None

parts = hostname.split('.')
if len(parts) > 2:
    return f'*.{".".join(parts[-2:])}'  # e.g., *.example.com
return hostname

2. Modified get_normal_form Method

   The get_normal_form method now includes the root domain in the normal form of the report.

   This ensures that reports with the same root domain are considered duplicates.

def get_normal_form(self) -> NormalForm:
"""Returns the normal form of the report, considering the root domain (wildcard)."""
root_domain = self.get_root_domain()
for reporter in get_all_reporters():
normal_form_rules = reporter.get_normal_form_rules()
if self.report_type in normal_form_rules:
normal_form = normal_form_rulesself.report_type # type: ignore
# Modify the normal form to include the root domain
if root_domain:
normal_form = (normal_form, ("root_domain", root_domain))
return normal_form
raise NotImplementedError(f"Don't know how to get normal form for {self.report_type}")

3. Added group_reports_by_root_domain Function

   This function groups reports by their root domain (wildcard).

def group_reports_by_root_domain(reports: List[Report]) -> Dict[str, List[Report]]:
"""
Groups reports by their root domain (wildcard).
"""
grouped_reports = defaultdict(list)
for report in reports:
root_domain = report.get_root_domain()
if root_domain:
grouped_reports[root_domain].append(report)
return grouped_reports

4. Added deduplicate_reports Function

   This function deduplicates reports by grouping them by root domain and selecting the best report for each group.

def deduplicate_reports(reports: List[Report]) -> List[Report]:
"""
Deduplicates reports by grouping them by root domain and selecting the best report for each group.
"""
grouped_reports = group_reports_by_root_domain(reports)
final_reports = []

for root_domain, report_group in grouped_reports.items():
    # Select the report with the highest score
    best_report = max(report_group, key=lambda r: r.get_score())
    final_reports.append(best_report)

return final_reports

Testing the Solution

To verify that the solution works, we created a test script (test_report.py) that tests the following:

get_root_domain Method:

    Verifies that the method correctly extracts the root domain (wildcard) from subdomains and URLs.

group_reports_by_root_domain Function:

    Verifies that reports are correctly grouped by their root domain.

deduplicate_reports Function:

    Verifies that reports are correctly deduplicated by selecting the best report for each root domain.

Problems Encountered During Testing

The script failed to run because Python couldn't find the artemis.reporting.base module.
When running tests in a Docker container, the script couldn't find the test_report.py file.

[Nazihbenbrahim]

Hi @kazet,

I hope you're doing well. I'm working on the Artemis project and facing multiple import-related issues. I've tried several approaches to resolve them, but I'm still stuck. Could you please help me figure out what's going wrong?
Problem Description

I’m encountering the following errors when trying to run my script (test_report.py):

ModuleNotFoundError: No module named 'artemis.reporting.base.normal_form'

ImportError: cannot import name 'ReportType' from 'artemis.reporting.base.report_type'

AttributeError: module 'artemis.reporting.base.reporters' has no attribute 'get_all_reporters'

These errors occur when I try to import modules in report.py.