Multilanguage tokenizer with language detection

[fmassot]

We want to support search on documents in different languages like common Latin languages (Eng, Fra, Deu...), Asian languages (Jpn, Cmn, ...), ...

To reach this goal, we need the following:

• a fast language detection algorithm as we don't want the detection phase to limit the indexing throughput. Cf whichlang repos.
• specific tokenizers for each language: for Latin languages, we could keep the current default tokenizer and have dedicated tokenizers for languages that are not Latin based (Chinese, Japanese, ...). There is jieba for Chinese and lindera for Japanese.
• one text field per language or one text field for all of them to store the tokens in the inverted index. Having one text field for all languages may be a good first step as managing several text fields adds extra complexity.

Last but not least, we should specify how to declare this multilanguage tokenizer in the index config before jumping into the code.
For example, a user should be able to define his custom tokenizer like this:

// index_config.yaml
tokenizers:
  multilanguage_jpn_cmn:
    default: default    // default tokenizer used
    cmn: jieba            // tokenizer used if cmn is detected
    jpn: lindera.         // tokenizer used if jpn is detected

[fulmicoton]

@fmassot Removing from 0.6. It would be nice to land this dangling PR though.

[fmassot]

Yea this is important to keep that for the 0.6. I will resume the work soon as the end user will test it very soon.