Fallback to binary search in suggestions search not working anymore?

[benoit74]

We have tests in python-scraperlib around the suggestions and full-text searches.

One of them is now failing with python-libzim 3.7.0 (linked to libzim 9.3.0):

https://github.com/openzim/python-scraperlib/blob/fef63f81fdb9dd6d2a5e17d9c8785e3fd22665e9/tests/zim/test_indexing.py#L114-L144

Test is failing on this line:

    assert "welcome" in list(reader.get_suggestions("brain"))

Same test with python-libzim 3.6.0 (linked to libzim 9.2.3-2) is OK.

This test specifically target the behavior of the libzim when there is no front article, and hence supposedly no Xapian suggestions index.

According to the comment in the test and linked issue, in such a case the libzim is supposed to fall back to binary search on titles in such a case, but it doesn't work as expected anymore.

I strongly suspect the issue is in libzim 9.3.0, since python-libzim is just a wrapper when it comes to suggestions. There is few lines of codes in python-scraperlib to automatically set the end value when fetching results, but I tried to bypass this code by setting an end value myself and it does not help.

[kelson42]

@veloman-yunkan Can you please fix this regression and ensure with have proper automated test at libzim?

[rgaudin]

We've also lost Archive.get_entry_by_title() which relies on the same Archive::getEntryByTitle(). We dont test this properly in pylibzim: openzim/python-libzim#223

[benoit74]

Please confirm this is only an issue at reading time, i.e. we are good to go with 9.3.0 to create ZIMs.

[benoit74]

@veloman-yunkan @kelson42 we need a quick feedback on my question above, this has very significant consequence in terms of release management of scrapers. I would like to release multiple scrapers in the coming days, preferably with latest libzim 9.3.0 if possible.

Fixing this regression is also important but probably a bit less urgent if it only impact "readers"

[veloman-yunkan]

This test specifically target the behavior of the libzim when there is no front article, and hence supposedly no Xapian suggestions index.

According to the comment in the test and linked issue, in such a case the libzim is supposed to fall back to binary search on titles in such a case, but it doesn't work as expected anymore.

Debugging shows that libzim indeed falls back to binary search on titles in this case, however it searches in listing/titleOrdered/v1 which only contains titles of front articles, i.e. is empty.

[benoit74]

Am I correct interpreting this is a consequence of #800?

Looks like the old "Title index v0" was not that useless in the end ^^

Thanks for the debugging / analysis. Now we need to make a decision on what's next.

Are there use cases where we do not have a Xapian suggestion "Title index" but do have content in the "Title index v1"? Because we've disabled the "Title index" creation but still have front articles? is that even possible today?

[rgaudin]

It's a behavior change so a decision is mandatory but I think the new behavior is aligned with what we've documented: suggestion is based on FRONT_ARTICLES.

I am more concerned about openzim/python-libzim#223 because from its name, Archive::getEntryByTitle(title) does not imply any dependency to FRONT_ARTICLES and the docstring doesn't mention anything.

[benoit74]

Documentation at https://libzim.readthedocs.io/en/latest/usage.html#searching-for-suggestions and https://libzim.readthedocs.io/en/latest/api/classzim_1_1Archive.html#_CPPv4NK3zim7Archive11findByTitleENSt6stringE doesn't clearly imply anything regarding FRONT_ARTICLES from my PoV, so it must probably be clarified / updated.

[veloman-yunkan]

Am I correct interpreting this is a consequence of #800?

I am not sure. I didn't debug how it worked with libzim 9.2.3-2, but to the best of my understanding it still should have used listing/titleOrdered/v1.

[veloman-yunkan]

Am I correct interpreting this is a consequence of #800?

I am not sure. I didn't debug how it worked with libzim 9.2.3-2, but to the best of my understanding it still should have used listing/titleOrdered/v1.

I was wrong. Before #949 listing/titleOrdered/v1 was not created if there were no front articles: https://github.com/openzim/libzim/pull/949/files#diff-33ba5fb8a510baa5046ced1b27724ce29e1a49174c1da11fab40850bba67166fL92-L94