New feature: sort results by relevance

[robert7]

Use case: if you have many notes, for many searches very often like 20 or even more results are found. The idea is to do some kind of relevance evaluation and sort "likely more relevant" hits towards beginning. Evaluating relevance is "rocket science", but I found out that quite simple algorithm does big improvement in daily usage.

I already implemented this in nixnote and did like this: if the word from search term, is found in title the result gets +1 point, if its found in tag, it gets +1 point, if the result hit note is marked with "important" tag, it gets +3 points (this is "manual override" - sometime works sometimes doesn't). So marking with "important" makes it some kind of "favorite" - but it only shows, when its a search hit. So you can thus have many hidden "favorites", which show on first places whenever they are search hit.

Then notes are sorted by count of points DESC and by second criteria (for me always "last updated"). The notes with "points" get some visual hint (I just used bold font for first version).

For some of the searches it works better, for some its not much difference, but overall its quite improvement (and the algoritm can be made better later - this is just first try). For this to work reasonably good, you need to fill meaningfull note titles.

The feature can be enabled/disabled in preferences ... if someone wouln't like (or just by selecting the sort criteria - if relevance field is not part of selected sort, it would be evaluated, or something like that).

Here is some example random searches I did, but there are cases where it performs better..

--
..this is one of the 2 "must have" features, which I would like to add. I don't yet have implementation plan, but I will do one, after I finish with the 3 UI tasks. Any hints/suggestions are welcome. I've been missing this feature in Evernote clients for years, to be honest I don't understand, why they don't add it. And it also one of the reasons why I decided to pick some open source client and add this myself.

[robert7]

Here some screenshots in "debug mode" where the relevance column is shown. Here first 3 hits got 2 points for finding both search terms in title + the first one is marked "important" got 3 additional.
Actually the first one is which I use most often.. There are notes found which are completely irrelevant to what I want to find, but now at least those are ordered towards bottom.

[d1vanov]

Thanks for the suggestion, I understand the idea and use case and I saw how you implemented it in Nixnote2. Regarding the implementation of such a thing in Quentier, I don't have a clear vision how to do it properly yet. The only thing I'd like to ask you about for now is please don't start coding this before we both agree on the implementation plan - this thing is complex enough to be properly designed before the actual implementation.

[robert7]

The only thing I'd like to ask you about for now is please don't start coding this before we both agree on the implementation plan

yes.

[robert7]

Basic implementation plan:
the note is currently searched for in few various sources, the basic "relevance weight" will be based on, where the record was found.
Sources:

1. "FROM NoteFTS WHERE contentListOfWords"
2. "FROM NoteFTS WHERE titleNormalized"
3. "FROM ResourceRecognitionDataFTS WHERE recognitionData"
4. "FROM NoteTags"
   Currently the SQL query is constructed that way, that source info is lost. The idea is to rewrite the query to retain this info, the calculate weight. Weight will be used in note sorting.

This will involve modification of LocalStorageManagerPrivate::noteSearchQueryContentSearchTermsToSQL(). Depending on whenever you want to be able to construct the query in both ways (with and without relevance info) a parameter would be added.

As the new query will not be distinct per localUid, deduplication will be handled in LocalStorageManagerPrivate::findNotesWithSearchQuery() e.g. using QSet. Calculated weight will be stored in "Note" and can be used for sorting.

Original query example:

New query example - the values 1,2,4 identify the source where the records was found:

The new query will be probably a bit slower and few more records will be read, but I believe it will be negligible in reality and that added benefit of better result will far over perform it.

I'm using the new sort in nixnote for few weeks and I would never want the old "quasi random" order back,

[d1vanov]

Thanks, I'll look into your suggestion in details later, for now I just wanted to say I don't think it's a good idea to store weights right inside notes, I would instead return from the method processing the search query the list of notes and their associated weights.

[robert7]

I would instead return from the method processing the search query the list of notes and their associated weights.

OK, that would be also possible. But we would probably need to change the interface.

Btw the plan doesn't 100% correspond to the original description. Its a bit simpler. For the first version there is not "important" evaluation (may be added later, but i'm not sure if it is even needed). Then the proposed solution doesn't distinguish how many words were found in "title" (this is a bit degradation, but this also may be added later).

[d1vanov]

I finally found some time to review the idea and I actually have another idea: compute weights for each found note after it has been found by the original query.

In other words, the output of LocalStorageManagerPrivate::findNotesWithSearchQuery is a list of notes found by the search query. At this moment a separate piece of code can skim through notes and the query used to find these notes and assign weights to each note as it relates to the search query. Remember that the code we are talking about is located within a library - libquentier. Although I am not sure any app but Quentier would ever use this library, the goal of a shared library should be being flexible and allow clients to adjust the library's behavior to its needs. In this case we could make the note weighting function customizable by library clients although providing some sane default logic. Take a look at headers/quentier/local_storage/ILocalStorageCacheExpiryChecker.h and headers/quentier/local_storage/DefaultLocalStorageCacheExpiryChecker.h and see how these are used in headers/quentier/local_storage/LocalStorageCacheManager.h - these illustrate the concept I'm suggesting quite well. Although it might have been an overkill to make the cache expiry function customizable that way but whatever, it's done already.

Regarding the possibility to not use notes weighting - probably it could be useful to leave this option for users who might want to just sort the found notes by modification time or by any other sorting criteria used for notes list view. I'd probably make this note searching with weighting a separate method with adjustable weighting function. Internally this new method can just call the old method and then assign weights to the found notes. Then depending on the setting either of two methods can be called.

Also regarding to how this new sorting by relevance should be implemented on NoteModel side in Quentier: whenever the notes are searched via the note search query, the list of note local uids is set to NoteFilterModel. In the new approach we could pass there the map or hash of note local uids with corresponding note weights. Then NoteFilterModel would pass this list on to its source NoteModel which inside itself would copy weights from the passed in map to relevant note model items and ensure that weights for all other note model items (not found by the search query) are equal to zero. Then the note model is just sorted by relevance (the previous sorting criteria needs to be stored somewhere outside of NoteModel to restore it after we are done with the search query i.e. when it's cleared out).

At this point we are done - we have a note search query which has produced the list of notes for which we have computed some weights and sorted the notes list view by these weights. When the note search query is cleared, the weights for all note model items should be zeroed again and previous sorting criteria should be restored.

It seems to me this idea should be much simpler to implement than messing with SQL queries while achieving quite the same thing. What do you think?

[robert7]

In my proposal I get the useful hints for weights directly from database (by identifying the source where information was found). In you version, you first throw them out and then try to recompute them. Which I guess will be way slower in case where you have many hits (200 hits may not unusual).

Although I am not sure any app but Quentier would ever use this library

I think: if there would by any users of the library beside quentier app, you will know about them. And the library is open source, so if they would feel bound too much, they can add the additional abstraction interface themselves and make improvement PR.

So basically any effort spent on non yet existing users is a bit waste of time in my eyes. So I try to be pragmatic and do some balance between wasting time and between creating technical debt. So I don't propose dirty code, but also not overkill too nice solutions, which aren't needed yet.

[d1vanov]

Ok, performance is a valid point. I'll think about it some more when I have some time.