Feature: YARA on-access scanning via fanotify (Linux)

[mhristodor]

Use case

On Linux endpoints, there's no built-in mechanism to detect malicious files at the point of write or execution. Two high-value detection moments are currently unaddressed:

1. Webshell drop — a PHP/JSP/ASPX file written to a web-accessible directory (e.g. /var/www, /srv/http) by a process that shouldn't be writing there (web server, CGI handler, compromised app).
2. Webshell execution — the interpreter (php, python, perl) executing a file that matches a known-malicious YARA rule.

Without on-access scanning, detection depends entirely on scheduled scans or log-based correlation, which introduces latency and misses fileless variants dropped and deleted quickly.

Proposed solution

Implement YARA on-access scanning using fanotify, which is the standard Linux kernel interface for filesystem event interception. Two event types are relevant:

• FAN_CLOSE_WRITE — fires when a file opened for writing is closed; use this to scan newly written files before they become available for execution.
• FAN_OPEN_EXEC — fires when a file is about to be executed; use this as a secondary gate.

To minimize overhead:

• Apply scanning only on FAN_CLOSE_WRITE by default; exec scanning is optional/configurable since it fires on every binary invocation system-wide.
• Scope fanotify mounts to configurable path prefixes (e.g. /var/www, /tmp, /dev/shm) rather than watching the full filesystem.
• Add a file extension filter before invoking YARA (e.g. only scan .php, .py, .pl, .sh, .so, .elf — configurable).
• Add an inode/hash cache for recently scanned, unmodified files to avoid rescanning on repeated opens.
• Run YARA evaluation in a dedicated thread pool with a bounded queue; drop or log-only if the queue is saturated (non-blocking mode).
• Expose a blocking mode toggle (via fanotify's FAN_ACCESS_PERM) for environments that can accept the latency tradeoff in exchange for active prevention.

Alternatives considered

• Scheduled YARA scans — already implementable externally; doesn't catch short-lived drops or act at execution time.
• inotify — cannot intercept or block, only notify after the fact; also less suitable for exec events.
• eBPF-based hooks — more flexible but significantly more complex to implement and maintain; kernel version dependencies. Could be a follow-up.
• Hash-based detection — low value for webshells since they're trivially modified; YARA string/regex matching is more resilient.

Additional context

• fanotify docs: man7.org/linux/man-pages/man7/fanotify.7.html — FAN_OPEN_EXEC requires Linux ≥ 5.0; FAN_CLOSE_WRITE is available from 3.x.
• ClamAV's clamonacc uses fanotify in a similar pattern and is a useful implementation reference.

[Karib0u]

Thanks for opening this, I agree the webshell drop/execution use case is very relevant for Rustinel.

I’d like to approach this in phases. Since Rustinel already has Linux file telemetry and an existing YARA file scanning pipeline, the first step should probably be detection-only: queue YARA scans from selected Linux file create/modify events, scoped to configurable paths such as "/var/www", "/srv/http", "/tmp", and "/dev/shm", with extension filtering.

Fanotify still makes sense for a more complete on-access design, especially for stronger "FAN_CLOSE_WRITE" / optional "FAN_OPEN_EXEC" semantics, but I’d prefer to evaluate that as a second phase after the simpler alert-only path is working and benchmarked.

So overall: yes, this fits the roadmap, but I’d start with the existing Rustinel pipeline first, then consider fanotify for stronger guarantees later.