Need customized stop word set for LuceneTextIndex

[jasperjiaguo]

In LuceneTextIndexCreator we are now hardcoding the stop words for Lucene text index.

Arrays.asList("a", "an", "and", "are", "as", "at", "be", "but", "by", "for", "if", "in", "into", "is", "it", "no",
          "not", "of", "on", "or", "such", "that", "the", "their", "then", "than", "there", "these", "they", "this",
          "to", "was", "will", "with", "those"),

These words will get pruned out during the text index generation as well as filter (in StandardAnalyzer). The problem with this is in production we found users issuing queries like
SELECT ... FROM ignoreMe WHERE TEXT_MATCH(title, '"IT staff" OR "IT manager"')
as will actually give the result matching TEXT_MATCH(title, '"staff" OR "manager"'). This can be easily reproduced in TextSearchQueriesTest.
cc @Jackie-Jiang @walterddr @siddharthteotia @SabrinaZhaozyf

[siddharthteotia]

We need to discuss / evaluate providing stop words in the form of SPI / plugin as opposed to one of the following:

• Changing the existing hardcoded list which will work but may impact (index size will increase) everyone else using text index.
• Making it customizable via tableConfig (which will work in theory but will make TableConfig super verbose / bloated

[jasperjiaguo]

@siddharthteotia can we put delta in tableconfig like
"exclude": ["IT"], "include": ["foo"] ?

[jasperjiaguo]

pr: #9708
user doc: https://docs.pinot.apache.org/basics/indexing/text-search-support#text-parsing-and-tokenization
This is done currently only for lucene. Support for raw text index can be added similarly as a follow up.

[Jackie-Jiang]

Thanks for adding the user doc!