Phrase Suggester: Suggesting on very frequent words can cause request failures

[nik9000]

I have a stack trace that looks like:

Caused by: java.lang.IllegalArgumentException: Fractional absolute document frequencies are not allowed
    at org.apache.lucene.search.spell.DirectSpellChecker.setThresholdFrequency(DirectSpellChecker.java:182)
    at org.elasticsearch.search.suggest.phrase.DirectCandidateGenerator.drawCandidates(DirectCandidateGenerator.java:131)
    at org.elasticsearch.search.suggest.phrase.MultiCandidateGeneratorWrapper.drawCandidates(MultiCandidateGeneratorWrapper.java:52)

I do not have and cannot get the index that causes this failure. But it looks to me like the failure is caused by this series of events:

1. DirectCandidateGenerator#thresholdFrequency spits out a frequency that is bigger than Integer.MAX_VALUE. This looks to be possible using the default configuration for common words like "the" when the corpus is a couple of million documents and each document is large, like, say, as big as a wikipedia page.
2. We call DirectSpellChecker#setThresholdFrequency with that number. The JVM helpfully casts the long returned by step 1 into a float, losing precision but keeping the magnitude of the number largely intact.
3. Lucene attempts to validate that the float is either less than 0 or a whole number. The "is it a whole number" check looks like thresholdFrequency != (int) thresholdFrequency. That will consider floats that don't fit into ints as not whole numbers. Most of the time, anyway.

There is a work around: set "suggest_mode": "always". We'll skip the math and just pick 0 for the frequency. Which is both less than one and whole number so Lucene is quite happy with it.

It looks like we should either clamp the value to Integer.MAX_VALUE in Elasticsearch or Lucene should use something else to check for fractional numbers.

[elasticmachine]

Pinging @elastic/es-search-aggs

[jimczi]

It looks like we should either clamp the value to Integer.MAX_VALUE in Elasticsearch or Lucene should use something else to check for fractional numbers.

I think there is a discrepancy in our usage of the Lucene suggester. DirectSpellChecker#suggestSimilar uses the docFreq (which is an int) of the candidate to compare with the threshold computed by DirectCandidateGenerator#thresholdFrequency. However in Elasticsearch we compute the threshold from the totalTermFreq and not the docFreq so this explains why the value can be greater than Integer.MAX_VALUE. This also means that we filter some terms that should not be filtered because we don't use the same metric (docFreq vs totalTermFreq). I think we should "just" switch to docFreq when computing the threshold.

[nik9000]

I think regardless of how we compute the threshold we should clamp it to Integer values so we don't trigger this.