Increase default prefix weight when prefix is requested

[sjudd]

Have you considered raising the default prefix match weight when search is performed with prefix: true? I was surprised to find that prefix matches are weighted lower than fuzzy matches by default, but I'm sure you have a good reason (curious what it is though!). I was more surprised that prefixes are still weighted worse by default (as far as I can tell) when you specifically request a prefix search.

For my use case, I'm searching ICD10 codes. One example of where the default behavior is not what I expected is this search:

Query: Z8744 
Options: { fuzzy: 0.15, prefix: true, fields: [ 'id' ] } 
Results: [
  {
    id: 'Z8774',
    score: 6.080555298224293,
    terms: [ 'z8774' ],
    queryTerms: [ 'z8744' ],
    match: { z8774: [Array] },
    description: 'Personal history of congenital malform of heart and circ sys',
    clinical_group: 'NA',
    comorbidity_group: null
  },
  {
    id: 'Z8742',
    score: 6.080555298224293,
    terms: [ 'z8742' ],
    queryTerms: [ 'z8744' ],
    match: { z8742: [Array] },
    description: 'Personal history of oth diseases of the female genital tract',
  },
  {
    id: 'Z8544',
    score: 6.080555298224293,
    terms: [ 'z8544' ],
    queryTerms: [ 'z8744' ],
    match: { z8544: [Array] },
    description: 'Personal history of malig neoplasm of female genital organs',
  },
  {
    id: 'T8744',
    score: 6.080555298224293,
    terms: [ 't8744' ],
    queryTerms: [ 'z8744' ],
    match: { t8744: [Array] },
    description: 'Infection of amputation stump, left lower extremity',
  },
  {
    id: 'Z87440',
    score: 5.791005045927898,
    terms: [ 'z87440' ],
    queryTerms: [ 'z8744' ],
    match: { z87440: [Array] },
    description: 'Personal history of urinary (tract) infections',
  },
  {
    id: 'Z87441',
    score: 5.791005045927898,
    terms: [ 'z87441' ],
    queryTerms: [ 'z8744' ],
    match: { z87441: [Array] },
    description: 'Personal history of nephrotic syndrome',
  },
  {
    id: 'Z87442',
    score: 5.791005045927898,
    terms: [ 'z87442' ],
    queryTerms: [ 'z8744' ],
    match: { z87442: [Array] },
    description: 'Personal history of urinary calculi',
  },

The user is looking for codes that start with Z8744, but the non-prefix and less relevant fuzzy matches are weighted higher and are showing up first.

This is easy enough for me to fix for my use case, so no big deal. Thanks for open sourcing this, great library!

[lucaong]

Hi @sjudd ,
you are right that your example leads to confusing results. I think custom weights are necessary in your case.

By default, MiniSearch assigns the following weights to fuzzy and prefix search (exact match has an implicit weight of 1): weights: { fuzzy: 0.45, prefix: 0.375 }. The weight scales down with distance, but prefix weight scales down more slowly to account for the fact that prefix matches stay relevant even when a short prefix matches a long term.

This causes single character typos to be scored better than prefixes, but fuzzier matches to score worse than most prefix matches. These defaults work well for many applications, but it's hard to come up with defaults weights that work for every case. In your case, even single character typos should score lower than most prefix matches. Luckily, it's easy to customize them for your application:

const miniSearch = new MiniSearch({
  searchOptions: {
    weights: { fuzzy: 0.4, prefix: 0.5 } // customize these values so they work well in your case
  },
  // your options here...
})

[lucaong]

I will close this issue, because there is a mechanism to customize the weights, and because tweaking the defaults would have more costs than benefits: it could break (or at least change the behavior of) existing applications relying on defaults, and would improve only specific cases. I hope that this issue provides some guidance for overriding the defaults for applications that need to do so.

@sjudd thanks for your report, what you say is correct in your case, I just think that changing the defaults has more costs than benefits.