set the token size limit and align it with lucene

[tarzanek]

since I cannot reopen #4170

I am filing a new bug

above bug was not fixed, since you only upgraded jflex
BUT
you need to go to respective analyzers that used the workaround , or even to ALL analyzers and set the

%token_size_limit (<identifier>|<number>)

and align it with current lucenes limits (check their tokenizers, I think they also use jflex)

@ginoaugustine FYI