EQL: Remove case_sensitive option

[costin]

Currently in EQL, requests allow a case_sensitive option for specifying whether string equality is case sensitive or not, as well the behavior for some functions. Initially this was introduced to determine the existing behavior and because the actual implementation requires scripting which is not ideal performance wise.

As Elasticsearch now supports such queries, this setting should be removed and EQL be always insensitive, where applicable.
This means:
operators: == and !=. Currently >=, <=, < and > are not supported
functions: startsWith, endsWith, stringContains, between, match, wildcard (note some regex might not fully support this, requiring the user to change the regex. This is considering missing functionality and will be added asap once the missing bits land in ES).

[elasticmachine]

Pinging @elastic/es-ql (:Query Languages/EQL)

[rw-access]

I think we still need the toggle for the detection engine, since I would expect both case-sensitive and case-insensitive queries to be widely used. @paulewing or @tsg can chime in more

[tsg]

Yes, from what I understood from @MikePaquette when this has come up before, we want equalities to be case-insensitive by default as that is most common, but there are also cases where we need case-sensitive equals. I'm not a fan of the global case-sensitive option because it introduces a "mode" that is external to the language and it doesn't let the rule author combine case-sensitive and case-insensitive comparisons in the same query.

Perhaps we can discuss this one live in the EQL meeting today.

[matriv]

I'm also against a global param that toggles case-sensitivity.

• It removes the flexibility to combine case-sensitive and case-insensitive comparisons in the same query
• Operators like >, <` that are case sensitive (exact) are not affected by this toggle, therefore removes the "global" effect of the toggle

So, I strongly prefer the per operator/function definition of the case-sensitivity.

[costin]

A belated update to this ticket after discussing it further:

1. the case-sensitive global flag is removed
2. if there will be a need in the future to add case sensitive equality, there is the option of a function exactEquals or potentially a different set of operators such as === (though their semantics in PHP and JS are different)
3. Equality operator == remains case insensitive, while the associated operators remains case sensitive.
4. Functions that require case sensitivity, will have a flag to turn-off this behavior. By default it is false meaning the functions, just like == are case insensitive.

In other words, the intent and scope of this ticket remains the same.