flood_stage not working with x-pack security

[nachogiljaldo]

Elasticsearch version (bin/elasticsearch --version): 6.3.2 (but the same symptoms appear with 6.0.1 and 6.4.0, although I did not debug them).

Plugins installed: [discovery-file, repository-s3]

JVM version (java -version):

java version "1.8.0_144"
Java(TM) SE Runtime Environment (build 1.8.0_144-b01)
Java HotSpot(TM) 64-Bit Server VM (build 25.144-b01, mixed mode)

OS version (uname -a if on a Unix-like system):
Linux 4efa9a510327 4.4.0-66-generic #87~14.04.1-Ubuntu SMP Fri Mar 3 17:32:36 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

Description of the problem including expected versus actual behavior:
cluster.routing.allocation.disk.watermark.flood_stage does not take effect when x-pack security is installed.
The logs show a message like:

[2018-08-24T08:07:51,796][WARN ][org.elasticsearch.cluster.routing.allocation.DiskThresholdMonitor] flood stage disk watermark [97%] exceeded on [1hFDDw9yRxGCIlfVRKuCCg][instance-0000000000][/app/data/nodes/0] free: 173.5mb[1.1%], all indices on this node will be marked read-only

but settings are not applied to the indices.

Steps to reproduce:

1. Create a cluster with x-pack security
2. Make sure cluster.routing.allocation.disk.threshold_enabled is enabled and configure cluster.routing.allocation.disk.watermark.flood_stage
3. You should see on the logs something like:

[2018-08-24T08:07:51,796][WARN ][org.elasticsearch.cluster.routing.allocation.DiskThresholdMonitor] flood stage disk watermark [97%] exceeded on [1hFDDw9yRxGCIlfVRKuCCg][instance-0000000000][/app/data/nodes/0] free: 173.5mb[1.1%], all indices on this node will be marked read-only

4. I would expect that my indices become read-only, but it does not happen

Provide logs (if relevant):
Did some debugging and it appears this exception is thrown:

ElasticsearchSecurityException[action [indices:admin/settings/update] is unauthorized for user [_system]]
    at org.elasticsearch.xpack.core.security.support.Exceptions.authorizationError(Exceptions.java:30)
    at org.elasticsearch.xpack.security.authz.AuthorizationService.denialException(AuthorizationService.java:575)
    at org.elasticsearch.xpack.security.authz.AuthorizationService.denial(AuthorizationService.java:553)
    at org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:158)
    at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$authorizeRequest$4(SecurityActionFilter.java:171)
    at org.elasticsearch.xpack.security.authz.AuthorizationUtils$AsyncAuthorizer.maybeRun(AuthorizationUtils.java:173)
    at org.elasticsearch.xpack.security.authz.AuthorizationUtils$AsyncAuthorizer.setRunAsRoles(AuthorizationUtils.java:167)
    at org.elasticsearch.xpack.security.authz.AuthorizationUtils$AsyncAuthorizer.authorize(AuthorizationUtils.java:149)
    at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.authorizeRequest(SecurityActionFilter.java:183)
    at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$3(SecurityActionFilter.java:161)
    at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:60)
    at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$writeAuthToContext$23(AuthenticationService.java:432)
    at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.writeAuthToContext(AuthenticationService.java:441)
    at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$handleNullToken$16(AuthenticationService.java:333)
    at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.handleNullToken(AuthenticationService.java:340)

and after some more debugging, it seems the culprit is https://github.com/elastic/elasticsearch/blob/v6.3.2/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/authz/privilege/SystemPrivilege.java#L17-L27 which apparently should include (at the very least) permissions for indices:admin/settings/update or even indices:admin/settings/* (provided it does raise any security concerns).

FWIW, I tried that (by compiling that class in 6.3.2 and substituting it) and it worked as expected after it

[elasticmachine]

Pinging @elastic/es-distributed

[elasticmachine]

Pinging @elastic/es-security

[bleskes]

@tvernum @jaymode this looks important to me. I bumped into it while catching up. Was this discussed?

[albertzaharovits]

Not sure how these "internal" actions should be handled but I can pick this up, if nobody else is bidding for it.

[tvernum]

@bleskes Sorry, no it dropped off the radar, but I agree it's important.

I think there are three changes we should make (in order from highest priority)

1. Change the _system role to grant the appropriate privilege
2. DiskThresholdMonitor.markIndicesReadOnly should have some sort or error handling/logging if the index settings change fails.
3. The AuthorizationService should have some logging level (DEBUG/TRACE) which logs all failures. IT should be possible to turn up the logging levels see these errors without needing to hack the code.

[jaymode]

I would be interested in an assert on access denieds for the system user to try to be more proactive about catching these than relying on logging

[bleskes]

@tvernum @jaymode +1 on my end. Are you guys going to work on it?

[tvernum]

I'll make sure someone picks this up in time for 6.5.0