Restrict repository indexing by glob match

[guillep2k]

EDIT: This PR mutated along the way; to get the current state of this PR, please jump to this comment.

Add an option in app.ini to restrict repository indexing by file extension:

[Indexer]
...
REPO_INDEXER_EXTENSIONS = go, c, cpp, h, idl, php, .
REPO_EXTENSIONS_LIST_EXCLUDE = false

It's not very powerful, but can help when indexes become polluted with derived files (e.g. lss, sym, map, etc.) so the search results are more relevant.

REPO_INDEXER_EXTENSIONS (default: empty) is a comma separated list of file extensions to include/exclude from the index. Only simple strings are supported (e.g. can't exclude ,v from CVS). A dot (.) matches files with no extension. An empty list means "include all files", disregarding the value of REPO_EXTENSIONS_LIST_EXCLUDE (i.e. the current behaviour).
REPO_EXTENSIONS_LIST_EXCLUDE (default: false) defines whether REPO_INDEXER_EXTENSIONS is a list of extensions to exclude rather than include.

I didn't update app.ini.sample because I thought it migth be confusing for the user, since the default value is empty, anyway.

This PR is related to #4831.

[lafriks]

Might it not be better to only have extensions not to index then. Probably as wildcards like *.lss, *.sym, *.map

[codecov-io]

Codecov Report

❗ No coverage uploaded for pull request base (master@10dfb44). Click here to learn what that means.
The diff coverage is 88.88%.

@@            Coverage Diff            @@
##             master    #7767   +/-   ##
=========================================
  Coverage          ?   41.83%           
=========================================
  Files             ?      482           
  Lines             ?    64583           
  Branches          ?        0           
=========================================
  Hits              ?    27018           
  Misses            ?    34095           
  Partials          ?     3470

Impacted Files	Coverage Δ
models/repo_indexer.go	70.15% <100%> (ø)
modules/setting/indexer.go	90.62% <78.57%> (ø)

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 10dfb44...e0fbcb7. Read the comment docs.

[guillep2k]

I suppose it depends on your particular case. I manage my company repository and there are projects ranging from VB6, C, C++, VB.Net, PHP, to Java and embedded. I still can't decide which method is more relevant to us. Lss, map, and sym are examples of files that definitely get in my way, but I'll probably decide for a list of known source code extensions to include rather than the opposite. There might be a lot of other use cases I can't think of, however.
I think that the REPO_EXTENSIONS_LIST_EXCLUDE flag is definitely useful, but there could be a REPO_EXTENSIONS_LIST_INCLUDE flag in its stead. The reason I decided for the former is that I could explain it better in fewer words 😅.

[lafriks]

Include list would be usually more hard to maintain and you will most probably don't even know if you are missing something that is not indexed. Include list would probably have very rare use case imho and that option would create more clutter and confusion than be helpful. But that is just my opinion :)

[guillep2k]

Would you rather have a REPO_EXTENSIONS_LIST_INCLUDE flag, then?

[lafriks]

Yes that would be less confusing

[guillep2k]

OK, syntax is now as follows:

[Indexer]
...
REPO_INDEXER_EXTENSIONS = go, *.c, .cpp, h, idl, php, .
REPO_EXTENSIONS_LIST_INCLUDE = false

REPO_INDEXER_EXTENSIONS (default: empty) is a comma separated list of file extensions to exclude from the index. An empty list means do not exclude files (i.e. the current behaviour).
REPO_EXTENSIONS_LIST_INCLUDE (default: false) if true, REPO_INDEXER_EXTENSIONS becomes a list of extensions to include rather than exclude.

REPO_INDEXER_EXTENSIONS now accepts *.txt, .txt and txt as synonyms. Dot (.) matches files with no extension.

[lafriks]

Probably you could use https://github.com/gobwas/glob for wildcard support

[guillep2k]

I've just added the gobwas/glob library for wildcard support. However, I think that power comes at the cost of added complexity.

In order to profit from the power of the glob patterns I'm using the full file path for matching (i.e. lib/drivers/uart.c vs. uart.c). This means patterns are not as intuitive as they could be. For instance, to match all the text files you need to use **.txt because the pattern *.txt only matches the files in the top level dir. On the other hand, it's possible to filter entire sub-directories using expressions like data/**, **/data/**.

I could have ignored the directory part of the path in the match and concentrate only on the actual file name, but then the library would be underutilized. Adding some examples to the docs would be very useful. I didn't want to pollute the config-cheat-sheet too much, so I kept it simple there.

To better reflect the effect of the filter, I've changed the names of the settings to REPO_INDEXER_PATTERNS and REPO_PATTERNS_INCLUDE.

Note: the last commit won't pass the build test because I don't know how to proceed with the added vendor. I've pushed just the code for your consideration.

PS: Sorry about the commit mess.

[guillep2k]

Added unit and integration tests. Please notice that the gobwas/glob library has a particular syntax, so my previous example:

REPO_INDEXER_INCLUDE = /project, /resources
REPO_INDEXER_EXCLUDE = .gitignore

should have been in fact written like:

REPO_INDEXER_INCLUDE = /project/**, /resources/**
REPO_INDEXER_EXCLUDE = **.gitignore

[guillep2k]

As this PR mutated quite a bit along the way, here is a summary of its functionality.

What this PR does

Adds an option in app.ini to restrict repository indexing by glob pattern matching using the gobwas/glob library:

[Indexer]
...
REPO_INDEXER_INCLUDE = project/**, resources/**
REPO_INDEXER_EXCLUDE = **.gitignore

It can help when indexes become polluted with derived files (e.g. lss, sym, map, etc.) so the search results are more relevant. It can also help reduce the index size.

REPO_INDEXER_INCLUDE (default: empty) is a comma separated list of glob patterns to include in the index. An empty list means "include all files".
REPO_INDEXER_EXCLUDE (default: empty) is a comma separated list of glob patterns to exclude from the index. Files that match this list will not be indexed. REPO_INDEXER_EXCLUDE takes precedence over REPO_INDEXER_INCLUDE.

Glob patterns as implemented by gobwas/glob are powerful and very fast, albeit a little complex. They work as follows:

• To match all files with .txt extension, no matter what directory, use **.txt.
• To match all files with .txt extension at the root level of the repository, use *.txt.
• To match all files inside resources/bin and below, use resources/bin/**.
• To match all files immediately inside resources/bin, use resources/bin/*.
• To match all Makefile files, use **Makefile.
• Matching a directory has no effect; resources/bin will not include/exclude files inside that directory; resources/bin/** will.
• All files and patterns are normalized to lower case.

[lafriks]

these last small things but otherwise great job! 🎉

[guillep2k]

Now gobwas is one of the vendors, so it makes more sense to use it here.

[lafriks]

make lg-tm work