[Feature request] Make results of initial scan available

[Shelim]

This is usecase for #339

Scenario:

• Platform with slow file-system access and a lot of files
• Need initial list of files as well as events on changes

Now I cannot obtain initial list of files from the API. I have to call separately walkdir which imply some drawbacks:

1. I need to rescan the whole file-system again
2. If one file is changed during rescanning, it changes can be lost (initial scan output will be provided later then changed event)
3. It imply memory cost for rescanning large file-system again
4. There may be some platform-specific inconsistencies in case symlink/hardlink. Initial set of files from notify may differ from walkdir output (unless notify is using walkdir internally, I haven't check up that)

[0xpr03]

There is no initial scan on windows. So this would be a special api for inotify / pollwatcher backend.

[Shelim]

Ah, I tricked myself the notify is doing initial scan as I only used pollwatcher. Okay, it seems legit now

[ShayBox]

This would be useful, I already use walkdir prior to notify (pollwatcher), this would prevent double scanning, speeding up my program startup, and reduce dependencies in my project.

[0xpr03]

How exactly would you like to access the initial scan ? Or do you want to access the live state too ?

One could want access to the full internal path cache, or alternatively provide a callback function that gets called for each path found during the initial scan.

[ShayBox]

I suppose whichever is easier, pollwatcher uses walkdir internally so exposing that would probably be easiest, but I'm not sure

Maybe an option to emit the first scan results to the existing event callback? That might be more work

[Shelim]

If we can change emitted event kinds, I believe the best way would be to add event callback kind for initial state. This way you can use exactly the same implementation mechanism you use for updated files

[0xpr03]

Could you try out #507 and give some feedback ? There is an example inside too.

[ShayBox]

Is there a way to optionally supply the same handler for both?

let mut watcher = PollWatcher::with_initial_scan(tx, notify_config, Some(tx))?;

EDIT: It seems the ScanEventHandler sends a ScanEvent which is a type alias for Result but the type alias for EventHandler excludes the Result, instead Result

struct Handler;

impl EventHandler for Handler {
    fn handle_event(&mut self, event: notify::Result<Event>) {
        todo!()
    }
}

impl ScanEventHandler for Handler {
    fn handle_event(&mut self, event: notify::Result<PathBuf>) {
        todo!()
    }
}

[0xpr03]

Hm yeah, I opted for putting the Result inside the ScanEvent type - maybe that's more confusing than anything (changing this for notify would be a semver break).

[0xpr03]

I left it like that and added some info to the ScanEventHandler docs.

[ShayBox]

Returning a result is fine, I just meant that normal event is wrapped in a result, while a scan event includes the result not just the pathbuf

Whats the purpose of the option parameter since the with variant is optional to use?

[0xpr03]

Whats the purpose of the option parameter since the with variant is optional to use?

Code unification

while a scan event includes the result not just the pathbuf

Yeah changing this to use the result in the normal event would be a major version change. But for the scan events it definitely makes sense to have this as one type.

[0xpr03]

Hm I could internalize this so you don't have to add a Some, why not.

[Shelim]

Yeah, since we already include with_initial_scan in the function name :-)

I looked at the example and it seems to be exactly what I needed. I will check the implementation in actual app in a few moments

[ShayBox]

This is what I came up with but 'Sender<PathBuf>' cannot be shared between threads safely

use std::{
    path::{Path, PathBuf},
    time::Duration,
};

use notify::{Config, Event, PollWatcher, RecursiveMode, Watcher};

fn main() {
    let (tx, rx) = std::sync::mpsc::channel();
    let mut watcher = PollWatcher::with_initial_scan(
        |event_result: notify::Result<Event>| {
            let Ok(event) = event_result else {
                return;
            };

            for path in event.paths {
                tx.send(path).unwrap();
            }
        },
        Config::default().with_poll_interval(Duration::from_secs(1)),
        Some(|path_result: notify::Result<PathBuf>| {
            let Ok(path) = path_result else {
                return;
            };

            tx.send(path).unwrap();
        }),
    )
    .unwrap();

    watcher
        .watch(Path::new("."), RecursiveMode::Recursive)
        .unwrap();

    loop {
        let path = rx.recv().unwrap();
        println!("path: {:?}", path);
    }
}

[0xpr03]

@ShayBox

fn main() {
    let (tx, rx) = std::sync::mpsc::channel();
    let tx_c = tx.clone();
    let mut watcher = PollWatcher::with_initial_scan(
        move |event_result: notify::Result<Event>| {
            let Ok(event) = event_result else {
                return;
            };

            for path in event.paths {
                tx_c.send(path).unwrap();
            }
        },
        Config::default().with_poll_interval(Duration::from_secs(1)),
        move |path_result: notify::Result<PathBuf>| {
            let Ok(path) = path_result else {
                return;
            };

            tx.send(path).unwrap();
        },
    )
    .unwrap();
   //[..]
}

See also the example. You need to move and clone() your tx once, as you need one per closure.

[ShayBox]

👍 It works for me, I can remove walkdir, rayon, and a few functions from my project now

[Shelim]

Looks good indeed, thank you!

[ShayBox]

Is there a planned release soon or can a release be made with these changes? I want to publish to crates.io but I can't use git dependencies on crates.io

[0xpr03]

I have a release planned, there are just some PRs I want to merge first