Devtools console throwing not found error on api call proxy?path=_aliases&method=GET

[liza-mae]

Kibana version: 6.4.0-SNAPSHOT
Elasticsearch version: 6.4.0-SNAPSHOT
Server OS version: Cloud
Browser version: Chrome
Browser OS version: Latest
Original install method (e.g. download page, yum, from source, etc.): Snapshots

Describe the bug:
Accessing devtools console is throwing a not found error, is this only happening on cloud, is this expected behavior ?

Steps to reproduce:

Navigate to devtools -> console

POST api/console/proxy?path=_aliases&method=GET
"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index","index_uuid":"na","index":"_all"}],"type":"index_not_found_exception","reason":"no such index","index_uuid":"na","index":"_all"},"status":404}

[imkarrer]

Took a quick peek and poked around a little bit. Here is what we found.

• Other ES versions do not seem to have this issue. Only 6.4.0-SNAPSHOT.
• Using the Kibana console and performing a GET on _aliases results in the same error
• Using the Kibana console and performing a GET on _cat/aliases results in the same error
• The cluster seems otherwise functional, it can index documents
• Using the ES API console from cloud results in the same error

I am unfamiliar with the expected behavior of the _aliases index, so I have some questions. I think I know, but I don't want to assume.

• What is the default behavior for the _aliases index? Should it exist by default?
• If it is not supposed to exist by default, then is Kibana responsible for ensuring its presence?
• If it is supposed to exist by default, where did it go? What information do you need to help determine what happened to the index?

[liza-mae]

Aliases don't exist by default, but with my local instance the command returns empty, not sure why on cloud it returns a different message of not found.

[imkarrer]

The really weird part is the disparity in the error message. The _aliases endpoint is being queried, but the error message cites the _all index as the missing index. But the _aliases index is listed when I GET _all.

[imkarrer]

@joegallo Could you lend us your eyes on this one. Maybe there is specific support for 6.4.0 that needs to be implemented for cloud? The issue appears to be isolated to the functional behavior of the ES cluster. @liza-mae states that the issue does not occur in a local environment with the same snapshot and only occurs on cloud.

[joegallo]

New aliases cannot be created

Can you double check that? I was able to create an alias.

[joegallo]

Anyway, my read of this is that it's a change in Elasticsearch (whether it's a bug or not I leave up to others).

6.3.0 and 6.3.1-SNAPSHOT will respond with an empty JSON object if there's no aliases, but 6.4.0-SNAPSHOT responds with a 404/error.

Maybe it's due to elastic/elasticsearch@38b1a5f? Seems likely to me.

In my tests, in order to see this behavior, you must have xpack.security.enabled: true and you must have no aliases.

But then it's just a matter of hitting /_aliases:

joegallo@radiant:~/Desktop/elasticsearch-6.3.1-SNAPSHOT $ curl -XGET -u"elastic_:jYWA3ScIznwdZ0MyAuoUbJr6" "localhost:9200/_aliases"
{}%

versus

joegallo@radiant:~/Desktop/elasticsearch-6.4.0-SNAPSHOT $ curl -XGET -u"elastic_:jYWA3ScIznwdZ0MyAuoUbJr6" "localhost:9200/_aliases"
{"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index","index_uuid":"_na_","index":"_all"}],"type":"index_not_found_exception","reason":"no such index","index_uuid":"_na_","index":"_all"},"status":404}%

Note: I needed to set up an elastic_ user with the superuser role via config/users and config/users_roles -- if you set up users using the native realm, that creates a .security-6 index and a .security alias for it, and then you don't have no aliases anymore. 😉

[joegallo]

/cc @javanna

[imkarrer]

New aliases cannot be created

Can you double check that? I was able to create an alias.

Ah, my description was imprecise. I can indeed create an alias. I was attempting to swap an index with an alias in one operation I took this snippet from the docs, and it failed. Apologies for the confusion.

PUT test     
PUT test_2   
POST /_aliases
{
    "actions" : [
        { "add":  { "index": "test_2", "alias": "test" } },
        { "remove_index": { "index": "test" } }  
    ]
}

Many thanks for confirming and identifying a commit that may have caused the issue @joegallo

[dakrone]

This is indeed caused by elastic/elasticsearch#31308 (though elastic/elasticsearch#31129 will have the same problem I suspect).

On 6.4.0-SNAPSHOT from today:

~/elasticsearch-6.4.0-SNAPSHOT λ curl -XGET -u"elastic_:jYWA3ScIznwdZ0MyAuoUbJr6" "localhost:9200/_aliases"
{}
~/elasticsearch-6.4.0-SNAPSHOT λ curl -X POST "localhost:9200/_xpack/license/start_trial?acknowledge=true"
{"acknowledged":true,"trial_was_started":true,"type":"trial"}
~/elasticsearch-6.4.0-SNAPSHOT λ curl -XGET -u"elastic_:jYWA3ScIznwdZ0MyAuoUbJr6" "localhost:9200/_aliases"
{"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index","index_uuid":"_na_","index":"_all"}],"type":"index_not_found_exception","reason":"no such index","index_uuid":"_na_","index":"_all"},"status":404}

And then the same snapshot, with elastic/elasticsearch@38b1a5f reverted:

~/elasticsearch-6.4.0-SNAPSHOT λ curl -XGET -u"elastic_:jYWA3ScIznwdZ0MyAuoUbJr6" "localhost:9200/_aliases"
{}
~/elasticsearch-6.4.0-SNAPSHOT λ curl -X POST "localhost:9200/_xpack/license/start_trial?acknowledge=true"
{"acknowledged":true,"trial_was_started":true,"type":"trial"}
~/elasticsearch-6.4.0-SNAPSHOT λ curl -XGET -u"elastic_:jYWA3ScIznwdZ0MyAuoUbJr6" "localhost:9200/_aliases"
{}

I'll open an issue on the Elasticsearch repo

[dakrone]

I opened elastic/elasticsearch#31516 for this