[ML] Adjacency weighting fixes in categorization

[droberts195]

In #1903 we changed dictionary weighting in categorization to give
higher weighting when there were 3 or more adjacent dictionary
words. This was the first time that we'd ever had the situation
where the same token could have a different weight in different
messages. Unfortunately the way this interacted with us requiring
equal weights when checking for common tokens meant tokens could
be bizarrely removed from categories. For example, with the
following two messages we'd put them in the same category but say
that "started" was not a common token:

• Service abcd was started
• Service reaper was started

This happens because "abcd" is not a dictionary word but "reaper"
is, so then "started" has weight 6 in the first message but weight
31 in the second. Considering "started" to NOT be a common token
in this case is extremely bad both intuitively and for the accuracy
of drilldown searches.

Therefore this PR changes the categorization code to consider
tokens equal if their token IDs are equal but their weights are
different. Weights are now only used to compute distance between
different tokens.

This causes the need for another change. It is no longer as simple
as it used to be to calculate the highest and lowest possible total
weight of a message that might possibly be considered similar to
the current message. This calculation now needs to take account of
possible adjacency weighting, either in the current message or in
the messages being considered as matches. (This also has the side
effect that we'll do a higher number of expensive Levenshtein
distance calculations, as fewer potential matches will be discarded
early by the simple weight check.)

[droberts195]

This is an example of an improvement from these changes. These two messages are very similar and intuitively should go in the same category. Previously they weren't because "PROXY" and "Session" are weighted differently.

[droberts195]

This is the example I saw on the Java side (while debugging elastic/elasticsearch#85872) that alerted me to the problem. Intuitively these messages are in the same category, but the old code would put them in different categories due to different weighting on the word "at".

[droberts195]

This is a case that's not as good following the changes. Previously it was nice that the extra weight on "Service" and "Source" meant that we differentiated services starting from sources starting. After the adjacency weighting changes we only carried on doing that in this unit test because the token "has" was considered not to match between the two messages (because it has the higher adjacency weighting in the second message but not in the first). Now that "has" is considered a match, that plus the massively high weighted verb "started" puts the two messages in the same category.

It's not ideal but I think the changes of this PR are more justifiable than what we had from #1903.

In the long term I think we should try to do the following:

1. Have the tokenizer do the weighting rather than the categorizer - this will facilitate 2 and 3
2. Only give higher weighting to adjacent dictionary words if they were separated by whitespace in the original message, not discarded tokens
3. Having decided we have a sufficiently long run of adjacent dictionary words, give higher weighting to all of them, not just the 3rd onwards - this will be fairer when one of the first two words is important (like "Source" vs "Service" in this case

[edsavage]

LGTM