[BUG] Escaped wildcard character in wildcard query not handled correctly

[HUSTERGS]

Describe the bug

When use wildcard query on wildcard field, raw * (and maybe include raw ?), is not handled correctly, while the same wildcard query can work well on keyword field. Possibly the escape logic is not implemented in opensearch/index/mapper/WildcardFieldMapper.java.

Related component

Search

To Reproduce

1. Create a simple index containing both wildcard and keyword type

   PUT escape_index
   {
     "mappings": {
       "properties": {
         "wild": {
           "type": "wildcard",
           "fields": {
             "keyword": {
               "type": "keyword"
             }
           }
         }
       }
     }
   }
   

2. Insert data with raw *

   POST escape_index/_doc
   {
     "wild": "* test *"
   }
   

3. search on both field with the same wildcard query
   Search on KEYWORD type:

   GET escape_index/_search
   {
     "query": {
       "wildcard": {
         "wild.keyword": {
           "value": "\\**"
         }
       }
     }
   }
   

   Result:

   {
     "took": 2,
     "timed_out": false,
     "_shards": {
       "total": 2,
       "successful": 2,
       "skipped": 0,
       "failed": 0
     },
     "hits": {
       "total": {
         "value": 1,
         "relation": "eq"
       },
       "max_score": 1,
       "hits": [
         {
           "_index": "escape_index",
           "_id": "xMi6opEB80UGmJnvgJSB",
           "_score": 1,
           "_source": {
             "wild": "* test *"
           }
         }
       ]
     }
   }
   

   Search on WILDCARD type

   GET escape_index/_search
   {
     "query": {
       "wildcard": {
         "wild": {
           "value": "\\**"
         }
       }
     }
   }
   

   Result:

   {
     "took": 1,
     "timed_out": false,
     "_shards": {
       "total": 2,
       "successful": 2,
       "skipped": 0,
       "failed": 0
     },
     "hits": {
       "total": {
         "value": 0,
         "relation": "eq"
       },
       "max_score": null,
       "hits": []
     }
   }
   

Expected behavior

get the correct results when wildcard query contains raw * and ? characters

Additional Details

No response

[msfroh]

That sounds like bug!

In particular, the methods findNonWildcardSequence and getNonWildcardSequence just scan through characters, splitting on ? or *. They don't check if those characters are escaped.

Also, I suspect that the logic in else case in

OpenSearch/server/src/main/java/org/opensearch/index/mapper/WildcardFieldMapper.java

Lines 433 to 450 in b71e547

	if (value.contains("?")) {
	Automaton automaton = WildcardQuery.toAutomaton(new Term(name(), finalValue));
	CompiledAutomaton compiledAutomaton = new CompiledAutomaton(automaton);
	matchPredicate = s -> {
	if (caseInsensitive) {
	s = s.toLowerCase(Locale.ROOT);
	}
	BytesRef valueBytes = BytesRefs.toBytesRef(s);
	return compiledAutomaton.runAutomaton.run(valueBytes.bytes, valueBytes.offset, valueBytes.length);
	};
	} else {
	matchPredicate = s -> {
	if (caseInsensitive) {
	s = s.toLowerCase(Locale.ROOT);
	}
	return Regex.simpleMatch(finalValue, s);
	};
	}

, which uses Regex.simpleMatch might not handle escapes properly either. Maybe we just go with the WildcardQuery.toAutomaton approach always, since it does handle escapes properly.

@HUSTERGS, are you willing to fix those methods? If not, I can take care of it (but probably not in time for the 2.17 release, which we expect to freeze today/tomorrow).

[HUSTERGS]

OF COURSE! @msfroh
bugfix is here #15737. And the original implementation cannot handle empty wildcard query strings, which will cause String index out of range on

OpenSearch/server/src/main/java/org/opensearch/index/mapper/WildcardFieldMapper.java

Line 479 in 1935650

terms.add(new String(new char[] { 0, currentSequence.charAt(0), currentSequence.charAt(1) }));

So I add a simple shortcut at the beginning of getRequiredNGrams to fix it all together