why isn't a gitignore pattern like `deep/deeper` matched at every level relative to the directory tree? #2824

fabiospampinato · 2024-05-27T18:31:56Z

fabiospampinato
May 27, 2024

Please tick this box to confirm you have reviewed the above.

I have a different issue.

What version of ripgrep are you using?

13.0.0

How did you install ripgrep?

brew install ripgrep

What operating system are you using ripgrep on?

macOS 12.5.1

Describe your bug.

I'm trying to understand how .gitignore files are handled.

Let's say I have the following structure:

<cwd>
├── .gitignore
└── deep
   ├── deep
   │  └── deeper
   │     └── bar.txt
   └── deeper
      └── foo.txt

And this is the content of my .gitignore file:

deep/deeper

According to my understanding of the spec that pattern is relative to the directory where the .gitignore file exists in, so it should match <cwd>/deep/deeper, so I'd expect <cwd>/deep/deeper/foo.txt to get ignored.

Yet if I run this command while at the <cwd>:

rg deep --files

I get the following output, basically the <cwd>/deep/deeper directory wasn't ignored:

deep/deep/deeper/bar.txt
deep/deeper/foo.txt

So I'm not quite understanding how that works in ripgrep, because:

if we are checking for the relative paths of files and directories, relative to the directory where that .gitignore file was found in, which I think is what the spec intended for it to happen, then presumably <cwd>/deep/deeper would be ignored, and <cwd>/deep/deeper/foo.txt wouldn't be listed by the rg command I'm running, so that doesn't seem to be the case here. Possibly for good reasons, otherwise if we are ignoring for example foo, but then the user runs a search inside <cwd>/foo, rg wouldn't find anything, which is most probably not what the user wanted, although in this case I'm excluding a directory inside the directory I'm searching into, so perhaps that should still be excluded?
if we are matching the rules in the .gitignore file against relative paths that are relative to our root search directory, which in this case is <cwd>/deep, then presumably <cwd>/deep/deep/deeper should have been excluded, but it isn't, so that's not what's happening either.

I guess the more specific questions are:

Are .gitignore files handled according to the spec?
If yes, then what strings is ripgrep actually trying to check if they are ignored by that .gitignore file?
If not, is this documented anywhere? I couldn't find more info about it.
Which hypothetical paths would a rule like "deep/deeper" actually ignore, according to ripgrep? And if the answer is "no paths" are we sure that's right?

What are the steps to reproduce the behavior?

mkdir -p test/deep/deeper
touch test/deep/deeper/foo.txt     

mkdir -p test/deep/deep/deeper
touch test/deep/deep/deeper/bar.txt

echo 'deep/deeper' >> .gitignore

cd test        
rg deep --files

What is the actual behavior?

deep/deeper/foo.txt is not ignored, and more generally ripgrep doesn't seem to be following the spec in ways that seem non-obvious.

What is the expected behavior?

deep/deeper/foo.txt to be ignored, and more generally handling .gitignore files to be less surprising, if it makes sense.

Answered by BurntSushi

May 27, 2024

Which seems its own problem, I wouldn't expect the presence of an empty folder named .git to change how .gitignore files are handled.

It does. From the man page:

       --no-require-git
           When this flag is given, source control ignore files such as
           .gitignore  are  respected  even  if  no  git  repository is
           present.

           By default, ripgrep will  only  respect  filter  rules  from
           source  control  ignore  files when ripgrep detects that the
           search is executed inside a source control  repository.  For
           example, when a .git directory is observed.

           This  flag  relaxes the default restriction. For example, it
…

View full answer

BurntSushi · 2024-05-27T19:01:10Z

BurntSushi
May 27, 2024
Maintainer

The spec says:

If there is a separator at the beginning or middle (or both) of the pattern, then the pattern is relative to the directory level of the particular .gitignore file itself. Otherwise the pattern may also match at any level below the .gitignore level.

Which means that if a separator is in the pattern, then it is matched relative to the directory of the .gitignore file. A pattern only matches at any level below the .gitignore file's directory when it doesn't contain a separator at the beginning or middle of a pattern.

Also, if ripgrep did work according to your understanding, then it should have ignored both foo.txt and bar.txt since both are in a deep/deeper directory. To achieve that behavior, just use **/deep/deeper.

Are .gitignore files handled according to the spec?

That's the goal but there can (and there are) bugs.

If yes, then what strings is ripgrep actually trying to check if they are ignored by that .gitignore file?

The path relative to the corresponding ignore file.

Which hypothetical paths would a rule like "deep/deeper" actually ignore, according to ripgrep? And if the answer is "no paths" are we sure that's right?

It would ignore deep/deeper relative to the corresponding .gitignore file. Which is what ripgrep does today:

$ cat .rgignore
deep/deeper

$ rg --files
test/deep/deep/deeper/bar.txt
test/deep/deeper/foo.txt

$ tree
.
├── deep
│   └── deeper
│       └── baz.txt
└── test
    └── deep
        ├── deep
        │   └── deeper
        │       └── bar.txt
        └── deeper
            └── foo.txt

8 directories, 3 files

Notice that I added deep/deeper/baz.txt to the directory hierarchy. And it is correctly ignored.

9 replies

BurntSushi May 27, 2024
Maintainer

... The directory tree in your screenshot doesn't match the one you gave. In your initial instructions, there is a test/.gitignore. But in your screenshot, you seem to have it at test/deep/.gitignore. This inconsistency should have been made clear when I shared my tree output. Please provide precise and correct reproduction instructions. EDIT: Sorry, my tree output didn't include hidden files, sigh. So that wouldn't have cleared it up. But there is still a seeming inconsistency between your screenshot and your instructions.

but the output changes if I execute rg deep --files, even though the location and content of my .gitignore file didn't change

It's possible you're hitting a bug and it's also possible that bug was fixed. The release of ripgrep you're using is almost 3 years old. You might want to consider upgrading. I don't support older versions of ripgrep.

fabiospampinato May 27, 2024
Author

... The directory tree in your screenshot doesn't match the one you gave. In your initial instructions, there is a test/.gitignore. But in your screenshot, you seem to have it at test/deep/.gitignore

Ah sorry, I messed up the shell commands when cleaning the text I had in the terminal 🤦‍♂️

To clarify, the structure was correct, this one:

<cwd>
├── .gitignore
└── deep
   ├── deep
   │  └── deeper
   │     └── bar.txt
   └── deeper
      └── foo.txt

The screenshot was correct, the .gitignore is indeed at test/.gitgnore, not at test/deep/.gitignore, it's just a bit hard to see the way that structure is rendered by vscode:

The commands were incorrect, we need to create the .gitignore inside the test folder, not outside of it:

mkdir -p test/deep/deeper
touch test/deep/deeper/foo.txt     

mkdir -p test/deep/deep/deeper
touch test/deep/deep/deeper/bar.txt

echo 'deep/deeper' >> test/.gitignore

And the command is executed inside the test folder:

cd test
rg deep --files

It's possible you're hitting a bug and it's also possible that bug was fixed. The release of ripgrep you're using is almost 3 years old. You might want to consider upgrading. I don't support older versions of ripgrep.

Oh you are right, sorry about that. The issue template on github suggests version v13.0.0 as an example of the version number, so I thought it was possibly recent, I should have double checked.

Under v14.1.0 (the latest that brew installed me) I see a different output now:

~/Desktop/test ❯ rg --version   
ripgrep 14.1.0

features:-simd-accel,+pcre2
simd(compile):+NEON
simd(runtime):+NEON

PCRE2 10.42 is available (JIT is available)
~/Desktop/test ❯ cat .gitignore
deep/deeper
~/Desktop/test ❯ rg . --files
./deep/deep/deeper/bar.txt
./deep/deeper/foo.txt
~/Desktop/test ❯ rg deep --files
deep/deep/deeper/bar.txt
deep/deeper/foo.txt
~/Desktop/test ❯

Basically now deep/deeper/foo.txt is not ignored when running rg . --files either, while I would expect it to be ignored with both commands instead.

Interestingly, if I initialize git inside the test folder I get a totally different output, while the content of my .gitignore didn't change:

~/Desktop/test ❯ git init     
Initialized empty Git repository in /Users/fabio/Desktop/test/.git/
~/Desktop/test ❯ cat .gitignore
deep/deeper
~/Desktop/test ❯ rg . --files   
./deep/deep/deeper/bar.txt
~/Desktop/test ❯ rg deep --files
~/Desktop/test ❯

And same if I just create an empty .git folder inside the test folder:

~/Desktop/test ❯ rm -rf .git
~/Desktop/test ❯ mkdir .git               
~/Desktop/test ❯ rg . --files   
./deep/deep/deeper/bar.txt
~/Desktop/test ❯ rg deep --files
~/Desktop/test ❯

Which seems its own problem, I wouldn't expect the presence of an empty folder named .git to change how .gitignore files are handled.

Hopefully this is clear correct and comprehensive enough, sorry for the wrong instructions I provided earlier and for using a version other than the latest one.

fabiospampinato May 27, 2024
Author

To further clarify I would expect the output for both rg . --files and rg deep --files to be basically this (either prefixed with ./ or not, the exact formatting doesn't matter):

deep/deep/deeper/bar.txt

Since the deep/deeper directory relative to my .gitignore file containing deep/deeper as its content should presumably be ignored by that, and the other deep/deep/deeper directory shouldn't be ignored by anything.

And I also would expect the presence of an empty directory named .git next to my .gitignore file to not change anything at all in the output of both commands.

BurntSushi May 27, 2024
Maintainer

Which seems its own problem, I wouldn't expect the presence of an empty folder named .git to change how .gitignore files are handled.

It does. From the man page:

       --no-require-git
           When this flag is given, source control ignore files such as
           .gitignore  are  respected  even  if  no  git  repository is
           present.

           By default, ripgrep will  only  respect  filter  rules  from
           source  control  ignore  files when ripgrep detects that the
           search is executed inside a source control  repository.  For
           example, when a .git directory is observed.

           This  flag  relaxes the default restriction. For example, it
           might be useful when the contents of a  git  repository  are
           stored  or  copied somewhere, but where the repository state
           is absent.

           This flag can be disabled with --require-git.

You should be using .ignore (generic ignore rules for anything using ripgrep's ignore crate) or .rgignore (ripgrep specific ignore rules) for non-git related patterns. .gitignore generally only makes sense in the context of .git directories. For example, ripgrep specifically knows not to respect .gitignore files that are from a different repository. (Because you might clone a git repository within a checkout of another repo. Many $HOME directories use this setup, for example.)

To further clarify I would expect the output for both rg . --files and rg deep --files to be basically this (either prefixed with ./ or not, the exact formatting doesn't matter):

OK, let's try this again. Copying from your revised instructions, but subbing out .gitignore for .rgignore as per above, I have this:

$ tree -a
.
├── deep
│   ├── deep
│   │   └── deeper
│   │       └── bar.txt
│   └── deeper
│       └── foo.txt
└── .rgignore

5 directories, 3 files

$ cat .rgignore
deep/deeper

And now the commands:

$ rg --files
deep/deep/deeper/bar.txt
$ rg --files deep
$

The output of the first command looks correct to me, and is I believe consistent with your expectation. The output of the second command does indeed appear wrong. I believe there are issues tracking this bug, but I'm not going to track it down now.

If you look at some of the issues surrounding ignore support today, you might get a more complete picture of the status quo. The status quo is that the ignore crate has always been a prototype, and it desperately needs to be overhauled. But practice breeds users, and there are a ton of programs written in Rust relying on the ignore crate. So I am in a situation where I don't want to make small changes (potentially risking breaking lots of folks) without rethinking the entire problem holistically. And I haven't had the inclination or time to do that yet. I'm busy with other projects.

Answer selected by BurntSushi

fabiospampinato May 27, 2024
Author

I see. Interesting that the presence of a .git directory matters 🤔

BurntSushi May 28, 2024
Maintainer

Because that's how you know a git repository likely exists. And .gitignore is git specific.

fabiospampinato May 28, 2024
Author

It makes sense, but I find it interesting because I'm not sure in what scenario one would create a .gitignore file that they would expect programs looking for .gitignore files to ignore, or like when "use this .gitignore file is you happened to put the directory inside a git repo, but not otherwise" is what one would want. Like it seems to make handling .gitignore files a bit less "deterministic", for lack of a better word.

BurntSushi May 28, 2024
Maintainer

As I said, you have to look for .git to know where one repository ends and another begins. Otherwise .gitignore files would impact filtering of nested repositories, which would be a departure from git.

And ripgrep specifically provides a way for you to opt out of this if you want to, via --no-require-git, as noted above.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

why isn't a gitignore pattern like `deep/deeper` matched at every level relative to the directory tree? #2824

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 1 comment 9 replies

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Select a reply

why isn't a gitignore pattern like deep/deeper matched at every level relative to the directory tree? #2824

fabiospampinato May 27, 2024

Please tick this box to confirm you have reviewed the above.

What version of ripgrep are you using?

How did you install ripgrep?

What operating system are you using ripgrep on?

Describe your bug.

What are the steps to reproduce the behavior?

What is the actual behavior?

What is the expected behavior?

Replies: 1 comment · 9 replies

BurntSushi May 27, 2024 Maintainer

BurntSushi May 27, 2024 Maintainer

fabiospampinato May 27, 2024 Author

fabiospampinato May 27, 2024 Author

BurntSushi May 27, 2024 Maintainer

fabiospampinato May 27, 2024 Author

BurntSushi May 28, 2024 Maintainer

fabiospampinato May 28, 2024 Author

BurntSushi May 28, 2024 Maintainer

why isn't a gitignore pattern like `deep/deeper` matched at every level relative to the directory tree? #2824

fabiospampinato
May 27, 2024

Replies: 1 comment 9 replies

BurntSushi
May 27, 2024
Maintainer

BurntSushi May 27, 2024
Maintainer

fabiospampinato May 27, 2024
Author

fabiospampinato May 27, 2024
Author

BurntSushi May 27, 2024
Maintainer

fabiospampinato May 27, 2024
Author

BurntSushi May 28, 2024
Maintainer

fabiospampinato May 28, 2024
Author

BurntSushi May 28, 2024
Maintainer