Change Cargo include/exclude rules to gitignore patterns

[behnam]

Based on discussion on this PR: #4256

Current status on master: Stage 1.1.

FAQ

How to migrate?

If you are receiving warnings about some include/exclude pattern (like xyz) matching new files in the future, and that's not what you want, please consider adding a leading slash to the pattern (xyz becomes /xyz). (Pending on PR: #4378)

Got problems with the Warnings?

If you are here because of a cargo warning that doesn't look right, or you cannot find a fix that works in both old (current) and new interpretations, please file an issue, with content of the package.include/package.exclude configs in your manifest file, as well as a copy of the warnings you have received.

Also, don't forget to put a link to this issue in your report, and mention the POC (@behnam) in the issue.

Goal

The current interpretation of Cargo's package.include and package.exclude configs is based on UNIX Globs, as implemented in the glob crate. Now we want these configs to work as similar to gitignore as possible. The gitignore specification is also based on Globs, but has a bunch of additional features that enable easier pattern writing and more control. Therefore, we are migrating the interpretation for the rules of these configs to use the ignore crate, and treat each include/exclude rule as a single line in a gitignore file.

Major features supported by the new (gitignore-like) interpretation are:

• Allow matching patterns fixed on the package root by using a leading slash (/) character.

  For example, exclude = [ "/data" ] will exclude file/directory named data under the root package directory. No other file/directory named data (like, /src/data) will be excluded.

• Matching all paths under a directory by only naming the directory itself.

  For example, you can have include = [ "/src" ] instead of include = [ "/src/*" ] or include = [ "/src/**" ].

• Support patterns with a trailing slash (/), which mean only match directories.

  For example, you can have exclude = [ "tables/" ] to exclude all directories named tables, with no effect on files.

Migration Stages

Stage 1. Implement new interpretation of include/exclude rules (the gitignore sytanx), compare the results with existing method, and warn if they don't agree on paths.

Also, error on any usage of negated patterns (rules staring with !), as they are treated literally at the moment, and will get a special meaning after the switch, which we want disabled by default. (! is the only new special character.)

Stage 1.1. Add leading-slash support to existing glob-based matching, to enable migration for patterns relying on matching the items in root. (Like "src/**/*", which should match other directories named src, except in the root.)

Stage 2. After at least one release and updating the docs, change the default to the new approach; and still warn if the new approach doesn't agree with the old approach.

Stage 3. In a later release, drop the old approach.

Possible Follow-ups

• Enable negated patterns (rules staring with !) for both include/exclude rules. This gives more control to users.

• Allow having both include/exclude rules at the same time. Not exactly related to this change, but easy and intuitive to do after these changes.

• If needed, update workspace.exclude to stay close to how package.exclude works.

Fixes #3578

[behnam]

Turns out ripgrep's ignore also has some issues in edge cases. I have added integration tests that reveal the bugs: BurntSushi/ripgrep#551

I'm going to expand the test and cover more cases and make sure ignore package is in good shape before we move forward here.