Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Is there a way to control the order of directory traversal? #1305

Open
ckp95 opened this issue May 1, 2023 · 2 comments
Open

Is there a way to control the order of directory traversal? #1305

ckp95 opened this issue May 1, 2023 · 2 comments
Labels

Comments

@ckp95
Copy link

ckp95 commented May 1, 2023

What version of fd are you using?
8.7.0

Hi, thanks for making such a great tool. One of the things I use it for is as a simple DIY file search-and-opener:

fd --exclude='blahblahblah' . $HOME | fzf | xargs xdg-open

The real script is more complicated than this, it has more fanciness, but that's the basic core of what it does.

What I've noticed is that almost of all of the files I search for are located in a handful of directories, perhaps a few dozen or so. These are like, my downloads folder, my folder full of saved pdfs, my folder for saved youtube videos, etc. It's one of those Pareto distribution things: 1% of the directories have 99% of the stuff I care about. But when I run my script, sometimes those directories take a while to appear in the fzf list (several seconds), because fd is traversing the filesystem in an arbitrary order, where the places it visits first are the ones unlikely to have the file I want.

What I would like to do is to give a parameter(s) to fd telling it to visit that directory before doing the full recursive search for everywhere else. Something like:

fd \
  --priority $HOME/downloads \
  --priority $HOME/documents/saved-articles \
  --priority $HOME/documents/books \
  ...
  --exclude='blahblahblah' \
  . $HOME | fzf | xargs xdg-open

Is this already possible to do? I could not see anything about it in the manual page. If it's not already possible to do, I would like to request it as a feature.

Pre-emptive notes:

  • This isn't a matter of fd searching for too much; I already give it a pretty substantial exclude pattern of directories that it shouldn't go into. It's already restricted to visiting only the places that I might want to open something from, in principle. So the ignore list is not the problem here, the problem is the order of traversal.
  • It also isn't about the order of the printed text output, at least not as such. I know that I can already pipe fd's output through sort or some other program to re-order the list how I want. But that still requires me to wait until fd visits those important directories in the first place (and actually longer, because sort can't be done in streaming fashion). This is about performance/latency, not presentation order.

Also, I don't really know how subdirectories of priority directories should be handled. e.g. suppose a directory layout like this:

~/videos
├── subtitles
│   ├── cat-video-1.srt
│   ├── cat-video-2.srt
│   └── cat-video-3.srt
├── cat-video-1.mp4
├── cat-video-2.mp4
└── cat-video-3.mp4
~/pics
├── camera-photos
│   ├── photo-2023-04-28_001.jpg
│   ├── photo-2023-04-28_002.jpg
│   └── photo-2023-05-01_001.jpg
├── memes
│   ├── funny-meme-1.jpg
│   └── funny-meme-2.jpg
├── abcd.jpg
├── efgh.jpg
└── ijkl.png

and I pass --priority=$HOME/videos --priority=$HOME/pics, it could conceivably list just the roots of the priority directories first, recursing into them later:

~/videos/cat-video-1.mp4
~/videos/cat-video-2.mp4
~/videos/cat-video-3.mp4
~/pics/abcd.jpg
~/pics/efgh.jpg
~/pics/ijkl.png
  ...
~/some/other/directory/some-file.txt
  ...
~/videos/subtitles/cat-video-1.srt
~/videos/subtitles/cat-video-2.srt
~/videos/subtitles/cat-video-3.srt
  ...
~/pics/camera-photos/photo-2023-04-28_001.jpg
~/pics/camera-photos/photo-2023-04-28_002.jpg
~/pics/camera-photos/photo-2023-05-01_001.jpg
~/pics/memes/funny-meme-1.jpg
~/pics/memes/funny-meme-2.jpg
  ...

or it might first recurse into subdirectories of the priority directories

~/videos/subtitles/cat-video-1.srt
~/videos/subtitles/cat-video-2.srt
~/videos/subtitles/cat-video-3.srt
~/videos/cat-video-1.mp4
~/videos/cat-video-2.mp4
~/videos/cat-video-3.mp4
~/pics/camera-photos/photo-2023-04-28_001.jpg
~/pics/camera-photos/photo-2023-04-28_002.jpg
~/pics/camera-photos/photo-2023-05-01_001.jpg
~/pics/memes/funny-meme-1.jpg
~/pics/memes/funny-meme-2.jpg
~/pics/abcd.jpg
~/pics/efgh.jpg
~/pics/ijkl.png
  ...
~/some/other/directory/some-file.txt

or it might give the root of the priority directories first, then recurse:

~/videos/cat-video-1.mp4
~/videos/cat-video-2.mp4
~/videos/cat-video-3.mp4
~/videos/subtitles/cat-video-1.srt
~/videos/subtitles/cat-video-2.srt
~/videos/subtitles/cat-video-3.srt
~/pics/abcd.jpg
~/pics/efgh.jpg
~/pics/ijkl.png
~/pics/camera-photos/photo-2023-04-28_001.jpg
~/pics/camera-photos/photo-2023-04-28_002.jpg
~/pics/camera-photos/photo-2023-05-01_001.jpg
~/pics/memes/funny-meme-1.jpg
~/pics/memes/funny-meme-2.jpg
  ...
~/some/other/directory/some-file.txt

I'm not sure which way is the best. For my use-case I think it would be the first one, but I can see how the others could be useful too, in different circumstances.

The way I am handling this at the moment is to run fd several times: first on each of the priority folders, then again on everywhere else, with the priority folders in the exclude list so they aren't visited twice. But this is really awkward to do, and probably not as performant as it could be.

@ckp95 ckp95 added the question label May 1, 2023
@tavianator
Copy link
Collaborator

Related: #734

The way I am handling this at the moment is to run fd several times: first on each of the priority folders, then again on everywhere else, with the priority folders in the exclude list so they aren't visited twice. But this is really awkward to do, and probably not as performant as it could be.

You can make that more performant (and more awkward) by running all the fds in parallel and merging their output:

stdbuf -oL -- paste -d'\n' \
    <(fd --search-path ~/videos) \
    <(fd --search-path ~/pics) \
    <(fd --search-path ~ -E /videos -E /pics) \
    | awk NF | fzf

stdbuf -oL keeps the output of paste line-buffered so it prints results as they come in, and awk NF removes empty lines from the output

@tavianator
Copy link
Collaborator

bfs can do this non-awkwardly too:

bfs ~/videos ~/pics ~ -unique

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants