norms file can become unexpectedly enormous [LUCENE-830]

[asfimport]

Spinoff from this user thread:

http://www.gossamer-threads.com/lists/lucene/java-user/46754

Norms are not stored sparsely, so even if a doc doesn't have field X
we still use up 1 byte in the norms file (and in memory when that
field is searched) for that segment. I think this is done for
performance at search time?

For indexes that have a large # documents where each document can have
wildly varying fields, each segment will use # documents times # fields
seen in that segment. When optimize merges all segments, that product
grows multiplicatively so the norms file for the single segment will
require far more storage than the sum of all previous segments' norm
files.

I think it's uncommon to have a huge number of distinct fields ? so
we would need a solution that doesn't hurt the more common case where
most documents have the same fields. Maybe something analogous to how
bitvectors are now optionally stored sparsely?

One simple workaround is to disable norms.

---

Migrated from LUCENE-830 by Michael McCandless (@mikemccand), resolved Jan 05 2012

[asfimport]

Doron Cohen (migrated from JIRA)

> One simple workaround is to disable norms.

You mean for some of the fields, using Fieldable's setOmitNorms().

For large indexes, I would think that most fields would be indexed with omit=true, except for one (content) or two (subject?) fields were length normalization and/or boosting are of importance. in such cases there would not really be a problem.

Consider the example that an index created for adding textual search to a database application, by mapping the index field names to the database "textual columns" names; if more than one table is indexed, but the textual column name happens to be different between the tables, then yes, - with that straightforward mapping there would be a waste - lots of unused bytes.

One work around for such applications could be to map the textual columns of all tables to a single textual field in Lucene, thuogh then they would have to filter by a table-name field (which they might do anyhow).

[asfimport]

Doron Cohen (migrated from JIRA)

> You mean for some of the fields, using Fieldable's setOmitNorms().

Oops, just noticed this was already suggested that in that for that user thread...

Anyhow, for that specific scenario seems omitNorms would be sufficient, but it won't help the db based example above.

[asfimport]

Grant Ingersoll (@gsingers) (migrated from JIRA)

Continuing a thread from IRC...

In Mahout, we have 1 dense vector representation along with a few sparse representations. In our case, we make users pick up front which representation they want based on what their data looks like and what algs they are running. The dense vector approach is pretty much just an array of whatever primitive, but the sparse ones are optimized towards either random access or sequential access. In Lucene's case, we probably could automatically pick an appropriate representation at IndexReader creation based on us keeping track of the density of norms for a given field.

The other thing to consider is we may want to allow people to separate out boosting from length normalization and allow each to be on or off.

[asfimport]

Robert Muir (@rmuir) (migrated from JIRA)

The other thing to consider is we may want to allow people to separate out boosting from length normalization and allow each to be on or off.

I think the first step is to move norm encode/decode float->byte out of Similarity (it does not belong here!)

In my opinion we should index individual statistics (boost, # of terms, etc). Ideally how these are encoded/decoded is part of the codec.

As far as how the raw stats are treated in scoring (such as if you want to combine #terms and boost into a single byte, and put it in a huge array, or do something else entirely), I think this belongs in Similarity. A lot of Similarities cant just use one single byte array for this, and others might not want to even
use bytes at all (this should be your choice, as you are making a tradeoff to lose precision intentionally for speed/RAM purposes).

If we shuffled things around like this, then for example you could have a Similarity that uses your sparse vectors instead of huge bytes for
"per-document normalization", and maybe it only cares about putting say the document boost in here. Its too limiting that we only have "huge byte[] or not"
and if people have ram issues (e.g. tons of fields) they are forced to both disable boosting and length normalization entirely.

Not really able to keep track of the realtime search issues, but it seems these things (static byte[]) are limiting there too.

By the way, what i described here is what Mike prototyped on the flexible scoring issue. I think it was good to prototype but I think it would be
much better to look at breaking that up into smaller digestible issues (e.g. adding the necessary stats to be indexed, making similarity per-field, ...)
so we actually make progress.

[asfimport]

Michael McCandless (@mikemccand) (migrated from JIRA)

As of 4.0, when norms are missing we drop norms for the entire field, unlike before when we invent a fake norm for documents missing that field or omitting norm for it.

Also, as of 4.0, you can now make a custom norm provider and custom similarity so if you really want to it's possible (in theory!) to have a sparse norms data structure...