Sparse index: optional skip list on top of doc values

[iverase]

Speaking to Adrien about how a sparse index would look like in lucene, he suggested that the sparse indexing does not need to be a new format bit an additional responsibility if DocValuesFormat.

The idea is to add an option to add a skip list on top of doc values and to expose it via the DocValuesSkipper abstraction, which has an API that is similar to what we're doing for impacts today. This provides a very simple index which can be very efficient when the index is sorted and the field belongs to the index sorting.

In order to implement it, we added a new flag in FieldType.java that configures whether to create a "skip index" for doc values. This flag is only allowed to be set on doc values of type NUMERIC, SORTED_NUMERIC, SORTED and SORTED_SET. Attempting to index other type of doc values with the flag set results on an exception.

This flag needs to be persisted on the FieldInfosFormat. This does not require a format change as we have some unused bit flags in Lucene94FieldInfosFormat that we can use.

We have changed the DocValuesFormat to generate the "skip index" whenever the flag is set. For this first implementation we went to the most basic implementation which consist in a skip list with just one level. In this level we collect the documents statistics every 4096 documents and we write them into the index. This basic structure already provides interesting numbers. I discussed with Adrien that as a follow up we should introduce more levels to the skip list and optimise the index for low cardinality fields.

In order to index a field with a skip list, we added static methods to the doc values field, for example NumericDocValuesField#indexedField(String name, long value) which will generated the right FieldType. In order to query it, you can use the existing NumericDocValuesField#newSlowRangeQuery(String field, long lowerValue, long upperValue). The generated query will use the skip index if exists by using the DocValuesRangeIterator.

Finally, here are some number I got using the geonames data set from lucene util.

The first test index the field called modified and adds the field as the primary sort of the index.

 Index LongField query LongField#newRangeQuery

  INDEX TIME: 42.604 sec
  INDEX DOCS: 12347608 documents
  INDEX SIZE: 12.402745246887207 MB
  QUERY TIME: 1157.7 ms
  QUERY DOCS: 6243379080 documents

Index LongField query IndexSortSortedNumericDocValuesRangeQuery

  INDEX TIME: 42.562 sec
  INDEX DOCS: 12347608 documents
  INDEX SIZE: 12.402745246887207 MB
  QUERY TIME: 662.6 ms
  QUERY DOCS: 6243379080 documents

Index Doc values skipping query SortedNumericDocValuesField#newSlowRangeQuery

  INDEX TIME: 38.927 sec
  INDEX DOCS: 12347608 documents
  INDEX SIZE: 11.800291061401367 MB
  QUERY TIME: 1072.5 ms
  QUERY DOCS: 6243379080 documents

This basic implementation is already faster that querying using the bkd tree. The IndexSortSortedNumericDocValuesRangeQuery is faster as it contains many optimisations but my expectation is that we can make this index as fast if not faster than this implementation.

The second test, we are indexing two fields and sorting the index using them; the countryCode as primary sort and the modified field as secondary sort. Then we execute the range queries on the modified field:

Index KeywordField, LongField query LongField#newRangeQuery

  INDEX TIME: 50.486 sec
  INDEX DOCS: 12347608 documents
  INDEX SIZE: 24.378992080688477 MB
  QUERY TIME: 1273.0 ms
  QUERY DOCS: 6243379080 documents

Index KeywordField, LongField query SortedNumericDocValuesField#newSlowRangeQuery

  INDEX TIME: 50.486 sec
  INDEX DOCS: 12347608 documents
  INDEX SIZE: 24.378992080688477 MB
  QUERY TIME: 13392.6 ms
  QUERY DOCS: 6243379080 documents

 Index Doc values skipping for both, query  SortedNumericDocValuesField#newSlowRangeQuery
  INDEX TIME: 44.127 sec
  INDEX DOCS: 12347608 documents
  INDEX SIZE: 16.09447193145752 MB
  QUERY TIME: 2975.0 ms
  QUERY DOCS: 6243379080 documents

In this case the query is slower than the BKD tree but still much faster than the brute approach. The advantage of the new query is that it does not need to build the big bitset that we might need to build with the BKD tree.

relates #11432

[jpountz]

I just did a full pass on changes again, and it looks good to me (disclaimed: I contributed to this branch). Would be good to have someone else take a look as well.

[ChrisHegarty]

This looks great. Just a few comments

[benwtrent]

I have minor comments.

The bulk of the implementation makes sense to me as a first step.

One weird thing is how we handle deletes. I am wondering about behavior when:

• All docs in a skipper are deleted. We skip the skipper-entry due to its maxDocID actually being deleted as well.
• Some of the docs deleted in a skipper. Our "min/max" at that point will not be accurate. We will still iterate the doc values in the skipper, which is OK. This just comes with the territory with deletes.

Are my ideas consistent with how we handle deletes?

I am pretty amazed that such a big change is only about 2k loc, many of which are just interface updates being populated.

[benwtrent]

Ah, I was wondering why we didn't use vint for the values, but now I see we keep track of the block size. Could you make this a constant?

[iverase]

The block size is actually 28. We read the first 4 bytes to compute the maxDocID and we skip the rest if it is not competitive.

I am hesitant to add a constant at the moment as this might change if we introduce levels.

[benwtrent]

Could we validate the skipper entry? My understanding is that length == 24*docCount right?

[iverase]

The validation would look like:

 assert length == 28 * (1 + ((docCount - 1) / SKIP_INDEX_INTERVAL_SIZE));

As before I am hesitant to add the validation at the moment as this might change if we introduce levels.

[easyice]

Do we need to do the same check in FieldInfos#verifySameSchema?

[iverase]

I think you are right but I wonder if we should optimize first FieldInfo.FieldNumbers. I open #13460 for consideration,

[iverase]

Added support on 7381364

[easyice]

Do we need to check the skip index in CheckIndex?

[iverase]

Added inital aupport on e782fef

[benwtrent]

This is an exciting first step!

🚀 🚀 🚀

[easyice]

Great work!

[iverase]

If there are no more comments, I will be merging this PR tomorrow.

[jpountz]

FWIW I'm trying to use #11432 as a meta issue for sparse indexing and started listing tasks that I think we (ideally) need to complete to be in a good state for 9.0.