New IndexReaderFunctions.positionLength from the norm

[dsmiley]

Description

Introduces org.apache.lucene.queries.function.IndexReaderFunctions#positionLength

Javadocs:

Creates a value source that returns the position length (number of terms) of a field, approximated from the "norm".

[rmuir]

I think the history is just that this norm can contain arbitrary value, which before was a suboptimal encoding into a single byte. There was a ValueSource that assumed it was a single byte, so that was moved to only work with TFIDF for backwards compatibility purposes.

Elsewhere, norm was extended and generalized to be opaque 64-bit value. Depending upon the Similarity's index-time computeNorm() implementation, it might not even be possible to decode to a float.

But the default encoding was also fixed to be practical, by @jpountz, whilst still using a single byte. So in practice all the built-in Similarities use the same encoding and can work with this: it just won't work if you extend Similarity to do something else.

Any confusion can be solved with documentation:

• should be clear that this only works, if your similarity uses the default implementation of computeNorm()
• don't think PositionLength is a good name, norm is not that (see discountOverlaps as an example).

Also I would ask if we really need this EMPTY instance: it would be good to keep polymorphism under wraps.

[dsmiley]

Thanks for the historical context!

I can definitely add more docs; I started with the bare minimum. Definitely need to emphasize a dependency on the default computeNorm formula! That documentation should also mention discountOverlaps.

I'm not married to the name; do you have ideas? If it has "term" in the name, people may be confused that the argument to the method is a term but it's just the field. Something like termLength would be confusing -- the length of what term? (no). Ultimately I like "position length" because it is the number of positions, the length, of the field. Using "length" in some way, I think, is likely to resonate with people on its use.

The code shows how EMPTY is needed (no norms). It mirrors the same for DoubleValues.EMPTY. I found it odd/surprising that it did not exist, and it's a common pattern I expect in Lucene. Is polymorphism really an issue here?

I anticipated possible doubt as to the placement of this. It's not a whole-index statistic, but it is related to the others here for there use in relevancy.

BTW I could imagine another interesting/useful utility method that takes a string (which is a query in practice), applies the index analyzer, counts the positions, and finally produces a constant and then build a constant LongValueSource from that. This would allow doing ~exact-ish matching of a query when combined with a phrase query targeting the field. Maybe nothing is needed at a Lucene level; it's a small amount of code that could be added at a higher level (like Solr).

[bruno-roustant]

Why not numTerms() instead of positionLength()?
Inside Similarity.computeNorm(), the value is named numTerms.

[dsmiley]

I'd expect a hypothetical IndexReaderFunctions.numTerms(field) to return the number of terms in the index for that field. That's not even close to what we want! "Length" should be a component of the name.

[jpountz]

What about calling it just "field length", since this is the length as computed for the purpose of length normalization?

[dsmiley]

fieldLength works for me. I'd like fieldPositionLength more as it characterizes the basis of the length (it's not characters). BTW some other methods on this class don't have "field" in the name yet take a field arg and so are a statistic about a field.

[dsmiley]

Would it make sense in this PR to add a Similarity.decodeNorm(long norm) returning an int of the field position length? It feels like the right thing to add.

[rmuir]

Please remove this method as it is impossible for someone to implement correctly if they customize just one field. The other method is per-field, this one is not.

[rmuir]

Also the name is wrong, there's nothing that requires this to be a position length. For some scoring methods it is something else such as the number of unique terms.

[dsmiley]

I should add the field name as an arg. The name is intentional -- if a Similarity can't decode the norm to a position length, it can throw UnsupportedOperationException.

[rmuir]

I don't agree with this being in the similarity api, sorry, that's too hacky.

[rmuir]

please rename this as there is nothing that requires the norm to be this. for example in some scoring methods it is the number of unique terms

[rmuir]

I think there is a high-level problem here, as i stated originally, that norm is not any position length. For example it may be based on FieldInvertState.getMaxTermFrequency() or FieldInvertState.getUniqueTermCount(), there are real scoring methods that use these approaches.

[dsmiley]

What name would you suggest then, Rob?
There's something to be said for choosing a name that's correct for the vast majority of cases, even if hypothetically a Similarity might do something else.

[rmuir]

I don't have any suggestion, I don't see the need for users to try to reimplement Similarity with valuesources.

[dsmiley]

The need is to incorporate a field's position length in a composable/flexible relevance formula. A LongValues is the way to do that. I understand a Lucene user could write a custom Similarity, which may be good for advanced search teams but most search teams would prefer to use use a flexible expression of some kind (hence a LongValues implementation at the Lucene layer) to trade some potential performance for expedience & simplicity (no-code).

Maybe a name like simply norm would work, documented to be whatever the Similarity says it is. Thus it wouldn't risk misadvertising itself as something it may not be. The method I added to Similarity would then be named just decodeNorm.

[rmuir]

Please don't add such changes to the similarity api that take us backwards. UOE is not an acceptable answer.

I think for now, it is best that such nonsense is contained to TFIDFSimilarity.

[dsmiley]

I'm trying to get where you're coming from...
The method on Similarity is there to keep the entire encoding information centralized to the Similarity so that the ValuesSource needn't pre presumptuous as to its encoding, and not even its meaning since renaming to simply "norm". How does that method "take us backwards"?

What is special about TFIDFSimilarity relative to this topic?

[github-actions]

This PR has not had activity in the past 2 weeks, labeling it as stale. If the PR is waiting for review, notify the dev@lucene.apache.org list. Thank you for your contribution!