Result for semantic search with query "[[HasLabel::BMD_StandardCU_V3]]" delivers random results #111

Activity

1. simontaurus commented on Nov 3, 2025

   

   simontaurus

   on Nov 3, 2025

   Contributor

   More actions

   Reference API request (now) returns stable result:

   https://healthbatt.projects01.open-semantic-lab.org/w/api.php?action=ask&query=[[HasLabel::BMD_StandardCU_V3]]|limit=1000&format=json

   Probably a data rebuild was running, leading to inconsistencies.
2. simontaurus

   closed this as completedon Nov 3, 2025
3. LukasGold commented on Aug 31, 2026

   

   LukasGold

   on Aug 31, 2026

   Contributor

   More actions

   The "random results" reproduce on healthbatt.projects01.open-semantic-lab.org and are not a data rebuild. Two causes, both measured.

   1. exists flickers server-side

   semantic_search only keeps results whose exists field equals "1" (https://github.com/OpenSemanticLab/osw-python/blob/main/src/osw/wiki_tools.py#L263). The field only ever takes '1' or '', but it is not stable. Eight identical [[Category:Item]]|limit=1000 calls:

   call 0: raw=1000  exists!=1 -> 6
   call 1: raw=1000  exists!=1 -> 2
   call 2: raw=1000  exists!=1 -> 6
   call 3: raw=1000  exists!=1 -> 6
   call 4: raw=1000  exists!=1 -> 6
   call 5: raw=1000  exists!=1 -> 2
   call 6: raw=1000  exists!=1 -> 2
   call 7: raw=1000  exists!=1 -> 6
   
   distinct `exists` values seen: {"'1'": 7964, "''": 36}
   pages reported non-existing in SOME call:  8
   pages reported non-existing in EVERY call: 0
   

   No page is consistently '', and the flips are perfectly correlated across all 8 pages (calls 0,2,3,4,7 agree; calls 1,5,6 agree). That bimodal pattern points at inconsistent state between backends or an SMW query-result cache, not at real deletions. Consecutive identical calls to semantic_search returned 998 or 994 titles.

   This needs fixing on the OSL instance side. The library can only make it visible.

   2. Silent truncation at the default limit

   The guess in the original report about the 1000 limit was right. SearchParam.limit defaults to 1000 and there is no pagination, so a larger match set is silently reduced to an arbitrary subset:

   [[Category:Item]] via |offset= pagination -> 2637 unique pages
   [[Category:Item]] via semantic_search     -> 1000 (no warning)
   

   Not a cause: sort order

   I checked whether adding |sort=|order=asc stabilises the truncated subset. It does not - the result hash still varies across identical calls, identically to the unsorted control. Appending a sort clause would also silently override a caller's own sort, since SMW honours the last one. So that is not a viable workaround.

   What changed

   #150 adds a warning when a query hits its limit and a warning naming how many results were dropped as non-existing, so neither failure mode is silent any more. It deliberately does not change the filter semantics or the default limit.

   Reopening is probably warranted for cause 1, since it is still live.