[BUG] query_string behavior using regex when shard failures occur due to too_complex_to_determinize_exception

[burnthm]

Describe the bug

Summary of analysis:

• able to see same behavior following repro steps below (replicated in AOS 2.17)
• use case is query_string query type which supports regular expression syntax
• I've translated the query_string example repro query into standard regex type and see the expected behavior where a 400 response is returned on the too_complex_to_determinize_exception
• I've compared the query profiles and they are nearly identical outside of a parent BoostQuery used in the query_string version of the request

What I understand is that the query_string is failing to catch shard failures, notably for the too_complex_to_determinize_exception error.

• I then tested if this was caused by allow_partial_search_results where silent errors may be ignored by the service in favor of returning some results to the client
  • relevant ES github issue: New query flag: allow_partial_results with default set to true. elastic/elasticsearch#27435; with related issue relevant to 200 response on potential failed shards: Should Elasticsearch return a non-200 response if there are shard failures? elastic/elasticsearch#18978
  • cluster setting default for search.default_allow_partial_results is true. this query parameter should override this cluster config for the specific query being executed.
  • this did not change the behavior of the query_string response, so it appears unrelated

It is especially worrisome that the too_complex_to_determinize_exception is neither caught during query execution or in the cluster error logs. Only after running the regex query type that faced the same actively thrown exception did I see it in the cluster's application logs.

Related component

Search

To Reproduce

1. Create test index

PUT test-regex
{
  "mappings": {
    "properties": {
      "f1": {
        "type": "keyword"
      }
    }
  }
}

2. Add sample document

PUT test-regex/_doc/1
{
  "f1": "value"
}

3. Send a request which fails silently with 200 response

GET test-regex/_search
{
  "query": {
    "bool": {
      "filter": {
        "bool": {
          "must": [
            {
              "query_string": {
                "query": "f1:/.*value.*|.*__two__.*|.*__three__.*|.*__four__.*|.*__five__.*|.*__six__.*|.*__seven__.*|.*__eight__.*|.*__nine__.*/",
                "analyze_wildcard": true
              }
            }
          ]
        }
      }
    }
  }
}

4. Send a request which succeeds with 200 response and returns sample document

GET test-regex/_search
{
  "query": {
    "bool": {
      "filter": {
        "bool": {
          "must": [
            {
              "query_string": {
                "query": "f1:/.*value.*|.*__two__.*|.*__three__.*|.*__four__.*|.*__five__.*|.*__six__.*|.*__seven__.*|.*__eight__.*/",
                "analyze_wildcard": true
              }
            }
          ]
        }
      }
    }
  }
}

5. Observe the too_complex_to_determinize_exception error on the silently failing query using the _validate API

GET test-regex/_validate/query?explain=true
{
  "query": {
    "bool": {
      "filter": {
        "bool": {
          "must": [
            {
              "query_string": {
                "query": "f1:/.*value.*|.*__two__.*|.*__three__.*|.*__four__.*|.*__five__.*|.*__six__.*|.*__seven__.*|.*__eight__.*|.*__nine__.*/",
                "analyze_wildcard": true
              }
            }
          ]
        }
      }
    }
  }
}

6. Convert the silently failing query_string using regex to an entirely regex query type and observe 400 response

POST test-regex/_search
{
  "query": {
    "regexp": {
      "f1": ".*value.*|.*__two__.*|.*__three__.*|.*__four__.*|.*__five__.*|.*__six__.*|.*__seven__.*|.*__eight__.*|.*__nine__.*"
    }
  }
}

Expected behavior

Response to query [3] from the reproduction above should return a 4xx response to the client, similar to the response observed when executing [6]:

{
  "error": {
    "root_cause": [
      {
        "type": "query_shard_exception",
        "reason": "failed to create query: Determinizing .*value.*|.*__two__.*|.*__three__.*|.*__four__.*|.*__five__.*|.*__six__.*|.*__seven__.*|.*__eight__.*|.*__nine__.* would require more than 10000 effort.",
        "index": "test-regex",
        "index_uuid": "hcEaXmFTSL-UJ86Pfdq0DA"
      }
    ],
    "type": "search_phase_execution_exception",
    "reason": "all shards failed",
    "phase": "query",
    "grouped": true,
    "failed_shards": [
      {
        "shard": 0,
        "index": "test-regex",
        "node": "X5jNaaOERuCxyStZa7CPhQ",
        "reason": {
          "type": "query_shard_exception",
          "reason": "failed to create query: Determinizing .*value.*|.*__two__.*|.*__three__.*|.*__four__.*|.*__five__.*|.*__six__.*|.*__seven__.*|.*__eight__.*|.*__nine__.* would require more than 10000 effort.",
          "index": "test-regex",
          "index_uuid": "hcEaXmFTSL-UJ86Pfdq0DA",
          "caused_by": {
            "type": "too_complex_to_determinize_exception",
            "reason": "Determinizing .*value.*|.*__two__.*|.*__three__.*|.*__four__.*|.*__five__.*|.*__six__.*|.*__seven__.*|.*__eight__.*|.*__nine__.* would require more than 10000 effort.",
            "caused_by": {
              "type": "too_complex_to_determinize_exception",
              "reason": "Determinizing automaton with 80 states and 319 transitions would require more than 10000 effort."
            }
          }
        }
      }
    ]
  },
  "status": 400
}

Additional Details

No response

[sandeshkr419]

[Search Triage] This particular case is where a wild-card field is ideal to use. Probably can think to mask the error better and make the case with/without validate consistent.

[jainankitk]

Thanks @burnthm for creating this issue with relevant repro and other details. I am wondering if in this particular case, the too_complex_to_determinize_exception is independent of the shard data/documents and just a function of the query itself.

Having said that, thinking more generally for different cases around search.default_allow_partial_results being set to true, the search execution flow should be below:

• Search request is fanned out to all the shards in the search request
• Every shard returns the corresponding results or the underlying error (too_complex_to_determinize_exception in this case)
• Coordinator determines if there is any successful shard execution? If not, respond with 4xx / 5xx error from one of the shards

We need to determine if the shard is correctly sending back the underlying error to the coordinator. If yes, the fix is required only on the coordinator, else we need to address both the coordinator and the data node

[jainankitk]

Running this locally even with allow_partial_search_results=false that overrides cluster level search.default_allow_partial_results setting, didn't cause the expected too_complex_to_determinize_exception:

% curl 'localhost:9200/test-regex/_search?allow_partial_search_results=false' -H 'Content-Type: application/json' -d '
{
  "query": {
    "bool": {
      "filter": {
        "bool": {
          "must": [
            {
              "query_string": {
                "query": "f1:/.*value.*|.*__two__.*|.*__three__.*|.*__four__.*|.*__five__.*|.*__six__.*|.*__seven__.*|.*__eight__.*|.*__nine__.*/",
                "analyze_wildcard": true
              }
            }
          ]
        }
      }
    }
  }
}'
{"took":42,"timed_out":false,"_shards":{"total":1,"successful":1,"skipped":0,"failed":0},"hits":{"total":{"value":0,"relation":"eq"},"max_score":null,"hits":[]}}% 

But able to see the exception by running it as regexp instead of query_string:

% curl localhost:9200/test-regex/_search -H 'Content-Type: application/json' -d '{ 
  "query": {
    "regexp": {
      "f1": ".*value.*|.*__two__.*|.*__three__.*|.*__four__.*|.*__five__.*|.*__six__.*|.*__seven__.*|.*__eight__.*|.*__nine__.*"
    }
  }
}'
{"error":{"root_cause":[{"type":"query_shard_exception","reason":"failed to create query: Determinizing automaton with 89 states and 115 transitions would require more than 10000 effort.","index":"test-regex","index_uuid":"9tEMjjgfT4Gfn28v8Zi8BQ"}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[{"shard":0,"index":"test-regex","node":"ZfuqdGNXTzGYQJ38l4QugA","reason":{"type":"query_shard_exception","reason":"failed to create query: Determinizing automaton with 89 states and 115 transitions would require more than 10000 effort.","index":"test-regex","index_uuid":"9tEMjjgfT4Gfn28v8Zi8BQ","caused_by":{"type":"too_complex_to_determinize_exception","reason":"Determinizing automaton with 89 states and 115 transitions would require more than 10000 effort."}}}]},"status":400}