feat!: search improvements

[frarees]

Motivation

DocFX's current state of the search functionality has some flaws:

• Searching through APIs can yield no results unless you type the complete symbol name
  • For example, if I have a page ManagedReferenceExample and I search for Managed I will get no results. Same goes for ManagedReference
  • This is specially important for API pages, since one of the reasons users want to use the search functionality is because they've forgotten the full name of a member
• Searching multiple terms doesn't narrow the search, but the opposite
• When I finish inputting my search query, I press Enter instinctively, which clears the search and reloads the page

What has changed

• New global metadata keys _searchIndexUseMetadata and _searchIndexUseMetadataTitle to opt-in
• New build step for MREFs: FillMetadata
  • Stores Title and Summary in ManifestProperties
• Search Index
  • Includes new property summary
  • Entries for MREFs can be loaded from metadata instead of HTML scrapping
• SearchIndexItem.Keywords holds different content for MREFs
  • Parses through the symbol's full name and creates permutations of the words found
    • ManagedReferenceExample -> Managed ManagedReference ManagedReferenceExample ManagedExample Reference ReferenceExample Example
  • Keywords is still present and it's used to store exclusively for keywords, and processed by the search engine
  • Summary is rendered by the themes (brief)
• Modern theme
  • Don't allow Enter/Submit on search
  • Search is additive: searching multiple terms uses AND instead of OR

Notes / TODO

• The regex used to create keywords, while accounting for most cases, is not perfect
• Default theme could use similar changes to modern theme, so UX is similar

[cdervout]

Hello
Any news about this PR ?

[frarees]

I'm done on my side (with some minor tweaks that could be done, see TODO).
Awaiting team's review.

I will push a merge against current main to keep it in sync.

[codecov]

Codecov Report

Attention: Patch coverage is 84.61538% with 10 lines in your changes missing coverage. Please review.

Project coverage is 79.36%. Comparing base (fe673ec) to head (3e393a4).
Report is 499 commits behind head on main.

Files with missing lines	Patch %	Lines
...c/Docfx.Build/PostProcessors/ExtractSearchIndex.cs	88.63%	1 Missing and 4 partials ⚠️
src/Docfx.Build/PostProcessors/SearchIndexItem.cs	44.44%	4 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #10113      +/-   ##
==========================================
+ Coverage   74.31%   79.36%   +5.04%     
==========================================
  Files         536      548      +12     
  Lines       23189    23686     +497     
  Branches     4056     4079      +23     
==========================================
+ Hits        17234    18799    +1565     
+ Misses       4853     3713    -1140     
- Partials     1102     1174      +72     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[frarees]

@yufeih any input on this? Is there anything else you'd like me to handle for this to get looked at / approved?

[yufeih]

Thank you @frarees

[Patrick8639]

@frarees After updating docfx with the PR and generating a new version of my documentation, the search feature is no more working!
Nothing is displayed in the results list, even when typing a full class name.

I have the following error in browser console:

search.ts:81  Uncaught TypeError: Cannot read properties of undefined (reading 'indexOf')
    at l (search.ts:81:32)
    at search.ts:111:29
    at Array.map (<anonymous>)
    at G (search.ts:107:41)
    at a (search.ts:94:7)
    at c.onmessage (search.ts:41:9)

Do you have an error about what's wrong?

[frarees]

@Patrick8639 can you set global metadata _searchIndexUseMetadata = true, and see if the issue is still there?

[Patrick8639]

@frarees It works better, as I now have results.
But not always: I have a class named Amount and, when I type Amoun, nothing is displayed.

Note that it would be great to have a default of true for the _searchIndexUseMetadata parameter, so it will work as soon as we set _enableSearch=true.

Also, is there documentation about this property?

[frarees]

@Patrick8639 I've tried locally using a development version (cloned repo) as well as the last release version (v2.78.3), with the provided sample project (seed/docfx.json, which has _searchIndexUseMetadata = false) and search works fine so far. Your project might be doing something specific I'm not catching, and I'll need to debug further on my side.

Are you using default theme or modern?

Let me know if you have further repro details I could use.

Note that it would be great to have a default of true for the _searchIndexUseMetadata parameter, so it will work as soon as we set _enableSearch=true.

That'd be ideal, although I left it opt-in to not break behaviour on current setups. In the future this might change.

Also, is there documentation about this property?

I don't think so, I can add the lines to the repo so next time the site is built they will be documented. Oversight on my side.

I have a class named Amount and, when I type Amoun, nothing is displayed.

Yes, the search engine setup could use further refinement to find words. Was it working before for this specific case, and is not after upgrade?

[filzrev]

I'm also encountered issues on my environment.
And it's resolved by clearing browser cache with Ctrl+F5.

[Patrick8639]

@filzrev

And it's resolved by clearing browser cache with Ctrl+F5.

I thought about it and tried: it doesn't work in my case.

[Patrick8639]

@frarees
Hi Francesco, thank you for your response.

Are you using default theme or modern?

Customized modern theme. This customization is about how the classes and members are displayed.
Perhaps I should try to compile the templates again.

Let me know if you have further repro details I could use.

I can give you the assemblies and all the files needed to compile the help file, including the custom template.

I have a class named Amount and, when I type Amoun, nothing is displayed.

Yes, the search engine setup could use further refinement to find words. Was it working before for this specific case, and is not after upgrade?

No, it wasn't working before, but I thought it will be after upgrade.

[frarees]

No, it wasn't working before, but I thought it will be after upgrade.

The way the search engine works is tricky. The main advantage I wanted to bring with the update is the ability to split words so that YouCanSearchThroughYourMembers more easily (here 'search members' would yield useful results for you). When it comes to incomplete words, the engine does algorithm-based stemming. But we need a bit more than than to make things work.

This is possibly not known (as in, I don't remember DocFX having this documented), but the search can use a variety of modifiers. Of special interest:

• Wildcards: Get*, *Listener
• Presence: +Get -Listener
• Fuzzy: Get~1 (matches words that differ in 1 character e.g., Set, Gets)
• Boost: Get^10 Listener (Get results are shown first, as they have better score)

Perhaps I should try to compile the templates again.

Specially important if you have a custom search.ts.

I can give you the assemblies and all the files needed to compile the help file, including the custom template.

Sure, you could create your own repo (or a branch if your project is already open?) and share a link here, and I'll take a look.

[Patrick8639]

Sure, you could create your own repo (or a branch if your project is already open?) and share a link here, and I'll take a look.

I will take a look later and send you something.

[PanzerFowst]

This is possibly not known (as in, I don't remember DocFX having this documented), but the search can use a variety of modifiers. Of special interest:

Wildcards: Get*, *Listener
Presence: +Get -Listener
Fuzzy: Get~1 (matches words that differ in 1 character e.g., Set, Gets)
Boost: Get^10 Listener (Get results are shown first, as they have better score)

Where is this documented?

Personally, I think the search engine would be more user friendly if it didn't have to use special modifiers on queries.

If I search for a method in my codebase, I want to be able to find it without having to remember exactly how it is spelled...and even then, I only get results usually based on namespaces and classes first instead of my method...

Are there any recommendations for how I could improve the search-ability?

[frarees]

Where is this documented?

Part of Lunr documentation, not documented officially on DocFX or the default or modern themes themselves afaik https://lunrjs.com/guides/searching.html

If I search for a method in my codebase, I want to be able to find it without having to remember exactly how it is spelled

Absolutely. The theme is not offering this yet.

Are there any recommendations for how I could improve the search-ability?

What I did for this PR is help split API members in words. Which improves searchability but it doesn't solve the problem completely.

Reading through Lunr documentation might give you ideas on how the theme could be improved to benefit from the features without having to manually specify the modifiers.

[pjohnst5]

Hi @frarees , how do you make a new docfx.min.js with your changes?

I'm wondering because I'm trying to change a few things in the search feature, but want to test it out in my browser first, do you know how I can do that?

[pjohnst5]

@frarees nvm, figured it out 😎

[pjohnst5]

Also wishing you well in coherence (looks pretty cool)

[Patrick8639]

@frarees nvm, figured it out 😎

It would be nice if you give the information for others...