CCS skip_unavailable drops remote results on 404

[kurtado]

ES version 6.3.1

When a search request on a CCS remote cluster attempts to query a remote index which doesn't exist, the response is different, depending on whether skip_unavailable is set to true. If skip_unavailable is false (or undefined), a 404 error is returned. However, when skip_unavailable is set to true, the remote results are simply dropped from the results, while the local results are returned.

Steps to reproduce:

Register the remote cluster without using skip_unavailable:

PUT _cluster/settings
{
  "persistent": {
    "search": {
      "remote": {
        "rcluster": {
          "seeds": [
            "server3:9300"
          ]
        }
      }
    }
  }
}

Index one doc in local cluster:

POST foo/_doc/
{
  "foo": "bar"
}

Index one doc in REMOTE cluster

POST /rfoo/_doc/
{
  "foo":"bar"
}

Query both indices, get results:

GET rcluster:rfoo,foo/_search

{
  "took": 62,
  "timed_out": false,
  "_shards": {
    "total": 10,
    "successful": 10,
    "skipped": 0,
    "failed": 0
  },
  "_clusters": {
    "total": 2,
    "successful": 2,
    "skipped": 0
  },
  "hits": {
    "total": 2,
    "max_score": 1,
    "hits": [
      {
        "_index": "foo",
        "_type": "_doc",
        "_id": "yVtG-GUBT7LcI9125HoM",
        "_score": 1,
        "_source": {
          "foo": "bar"
        }
      },
      {
        "_index": "rcluster:rfoo",
        "_type": "_doc",
        "_id": "hm9F-GUBOiyLU_wOMPWD",
        "_score": 1,
        "_source": {
          "foo": "bar"
        }
      }
    ]
  }
}

Query for a nonexistent remote index, and 404 is returned (same behavior as a purely local query)

GET rcluster:rfoo2,foo/_search

{
  "error": {
    "root_cause": [
      {
        "type": "index_not_found_exception",
        "reason": "no such index",
        "index_uuid": "_na_",
        "resource.type": "index_or_alias",
        "resource.id": "rfoo2",
        "index": "rfoo2"
      }
    ],
    "type": "index_not_found_exception",
    "reason": "no such index",
    "index_uuid": "_na_",
    "resource.type": "index_or_alias",
    "resource.id": "rfoo2",
    "index": "rfoo2"
  },
  "status": 404
}

Now, register the remote cluster with the skip_unavailable setting:

PUT _cluster/settings
{
  "persistent": {
    "search": {
      "remote": {
        "rcluster": {
          "seeds": [
            "server3:9300"
          ],
          "skip_unavailable": true
        }
      }
    }
  }
}

Now, query including the exact same remote index which does not exist, and now instead of a 404, the response silently ignores the missing remote index, but returns local results

GET rcluster:rfoo2,foo/_search

{
  "took": 4,
  "timed_out": false,
  "_shards": {
    "total": 5,
    "successful": 5,
    "skipped": 0,
    "failed": 0
  },
  "_clusters": {
    "total": 2,
    "successful": 1,
    "skipped": 1
  },
  "hits": {
    "total": 1,
    "max_score": 1,
    "hits": [
      {
        "_index": "foo",
        "_type": "_doc",
        "_id": "yVtG-GUBT7LcI9125HoM",
        "_score": 1,
        "_source": {
          "foo": "bar"
        }
      }
    ]
  }
}

Discussed with @jasontedor, who agreed that remote missing index should behave consistently as it does with a local missing index, and return a 404.

[jasontedor]

I indeed discussed this with @kurtado. A few comments: my understanding is that skip_unavailable is only for handling the case of a cluster that is not reachable at the time of query execution. However, we are using skip_unavailable to treat any error from the upstream cluster as success and provide an empty response. I think this is contrary to the intent of skip_unavailable and that we should indeed be treating these (e.g., an index not found exception on a remote cluster) as we treat them on the local cluster.

[elasticmachine]

Pinging @elastic/es-search-aggs

[javanna]

Agreed that the initial implementation of skip_unavailable was a bit naive. This was brought up initially (#27182 (comment)) but we kept things simple to start with. I would consider this a bug. I need to look into how to distinguish between errors that need to be ignored and errors that have to be thrown. I guess that apart from timeouts we should keep on ignoring 5xx errors, while all the others should be returned and cause the whole request to fail (as they do not have to do with availability of the remote clusters)?