Search - expose new Lucene option for case insensitive RegExp queries

[markharwood]

Lucene 9386 introduced an option for case insensitive matching of RegExp queries. This issue is to expose that capability to elasticsearch users.

One complexity with the feature is that it currently only supports ASCII characters and full support for Unicode has been left as a TODO. Rather than being a simple Boolean flag (eg case sensitive - yes or no?) there is a bitmask parameter which accepts only one flag at the moment - "ASCII_CASE_INSENSITIVE". In future the assumption is that we could use this same bitmask parameter and pass a new "(Unicode)CASE_INSENSITIVE" flag.

Another concern is in Lucene we renamed the previous "flags" constructor parameter to "syntax_flags" to make it distinct from the new "match_flags" parameter. Renaming constructor parameters works OK in Java because they are identified by position and type but in our REST api we rely on JSON field names so have to keep the existing flags parameter.
The question then perhaps is how best to expose the new case insensitive matching flag in the REST api?
Options are:

1. Add an "ASCII_CASE_INSENSITIVE" value option for the existing flags parameter
2. Add a new "match_flags" parameter with possible value option of "ASCII_CASE_INSENSITIVE"
3. Add a simple Boolean flag that can only accept one value (the flag name choice would be case_sensitive:false or case_insensitive:true). We'd use documentation to declare that only ASCII is supported currently.

[elasticmachine]

Pinging @elastic/es-search (:Search/Search)

[jpountz]

I'm not a fan of 1 because it creates issues with ALL since I don't think it should affect case sensitivity.

I have a slight preference for 3 because I fear that otherwise we would mostly be pushing complexity downstream. I'm running under the assumption that it wouldn't be a breaking change if we eventually made it handle unicode, which might be controversial.

[markharwood]

I'll opt for 3. As for the BWC concern I expect the behaviour change would be welcome. I assume the intent of asking for case_insensitive:true is always the same and the more able we are to fulfil that, the better.