Document Field Level Security on Frozen Tier not working correctly

[madisonb]

Elasticsearch version (bin/elasticsearch --version): 7.16.2 (running inside Elastic Cloud)

Plugins installed: []

JVM version (java -version): n/a (Elastic Cloud)

OS version (uname -a if on a Unix-like system): n/a (Elastic Cloud)

Description of the problem including expected versus actual behavior:

I seem to be running into an issue where Field Level Security throws a null exception when operating on frozen indices.

I have a simple ILM policy for my index that moves data from Hot to Frozen after 12 hours. Within that data set, I would like to grant access to all fields except for a few specific ones that I would like to remain internal only.

If I create a new user and grant them a custom role with field level security (allowing and denying specific fields), that user cannot search for anything beyond my hot data tier without getting the following exception back

"reason": "unsupported_operation_exception: null"

Within the data access role, If I disable Grant access to specific fields, the user can see and return results from the frozen tier.

I will note that in my current environment, this role also is using a Grant read privileges to specific documents templated query, however that does not seem to have an impact on this issue. I have tried to produce a working example below that does not involve that privilege.

Steps to reproduce:

1. Create a simple ILM policy that rolls data out of a hot index and into a frozen index

2. Index data into your ILM managed index so that you have both hot data AND frozen data within your cluster. If my ILM index alias was called pulse, my underlying indices are pulse-0001, pulse-0002, etc and the frozen indices look like partial-pulse-0001, partial-pulse-0002... etc

3. Create a new role that grants read access to you your desired indices, like below (I am using Kibana):

4. Create a new user, and assign them typical access to a kibana space and grant them the data role from step 3

5. In a new private browser, log in as your new user and validate they have access to your frozen tier data and hot tier data, by viewing the Discover panel and looking at a timerange that spans hot and frozen tiers. (24 hrs in my case, see below as an example)

6. Go back to the role you created as an admin, and check the box Grant access to specific fields. Deny a field in your data (see below as an example)

7. Back as your new user, refresh the page to see shard exceptions being thrown for all your frozen indices (even though my time range is still set to 24 hours, I get exceptions for my entire frozen tier)

Note in the screenshot above that my data is cut off arbitrarily, right near my frozen tier rollover line from my ILM policy

8. Investigate the exception further and you get the following

9. Clicking the tab for "Request" shows very normal request, and the "Response" tab looks like below:

10. From the command line, I can search the cluster easily if I use a simple count search on a hot tier index

curl https://user:pass@my-cluster.es.us-east-1.aws.found.io:9243/pulse-000252/_count
# returns
{"count":<real number here>,"_shards":{"total":1,"successful":1,"skipped":0,"failed":0}}

But if I try to do an operation on the whole alias that includes frozen shards, I get shard exceptions.

curl https://user:pass@my-cluster.es.us-east-1.aws.found.io:9243/pulse/_count
# returns 
{"count":<partial number here>,"_shards":{"total":248,"successful":14,"skipped":0,"failed":234,"failures":[{"shard":0,"index":"partial-pulse-000015","node":"XCRMYhdLR3KHuHxm74vlCg","reason":{"type":"unsupported_operation_exception","reason":"unsupported_operation_exception: null"}},{"shard":0,"index":"partial-pulse-000016","node":"9SNaA5L9TCqZ8l0BA39c1Q","reason":{"type":"unsupported_operation_exception","reason":"unsupported_operation_exception: null"}},{"shard":0,"index":"partial-pulse-000017","node":"XCRMYhdLR3KHuHxm74vlCg","reason":{"type":"unsupported_operation_exception","reason":"unsupported_operation_exception: null"}},.....

11. For sanity you can go back to your role configuration and uncheck "Grant access to specific fields" and run that _count command again:

curl https://user:pass@my-cluster.es.us-east-1.aws.found.io:9243/pulse/_count
{"count":<real number here>,"_shards":{"total":248,"successful":248,"skipped":0,"failed":0}}

and it works.

I have also tried combing through the built in roles for Elastic, as well as the built in index priviledges to see if there was anything related to the frozen tier specifically that causes this behavior, without much luck.

Provide logs (if relevant):

I have tried to comb the logs inside of Elastic Cloud but the UI does not seem to be surfacing this exception where I can find it.

[elasticmachine]

Pinging @elastic/es-security (Team:Security)

[albertzaharovits]

@madisonb Thank you for reporting problems with Elasticsearch!

Unfortunately I'm having trouble replicating the issue.
Can you please share more details about the role and the index settings and mappings?
Does it work if you query directly the partially mounted indices rather than the alias?

[albertzaharovits]

For the record, I have tried on a local 7.16.2 node, with the following settings:

path.repo: "/tmp/elasticsearch"
xpack.searchable.snapshot.shared_cache.size: "100mb"
xpack.license.self_generated.type: "trial"

the following commands:

# create index
curl -k -u 'elastic:password' -X PUT "http://localhost:9200/twitter/_doc/1" -H 'Content-Type: application/json' -d'
{
  "category": "click", "tag": "one"
}
'
curl -k -u 'elastic:password' -X PUT "http://localhost:9200/twitter/_doc/2" -H 'Content-Type: application/json' -d'
{
  "category": "click", "tag": "two"
}
'

# create repo
 curl -k -u 'elastic:password' -X PUT "http://localhost:9200/_snapshot/fsbackup?pretty" -H 'Content-Type: application/json' -d'
{
  "type": "fs",
  "settings": {
    "location": "/tmp/elasticsearch",
    "compress": true
  }
}
'

# create ILM policy
curl -k -u 'elastic:password' -X PUT "http://localhost:9200/_ilm/policy/ilmpolicy?pretty" -H 'Content-Type: application/json' -d'
{
  "policy": {
    "phases": {
      "frozen": {
        "actions": {
          "searchable_snapshot" : {
            "snapshot_repository" : "fsbackup"
          }
        }
      }
    }
  }
}
'

# manually apply the ILM policy to the index
curl -k -u 'elastic:password' -X PUT "http://localhost:9200/twitter*/_settings?pretty" -H 'Content-Type: application/json' -d'
{
  "index": {
    "lifecycle": {
      "name": "ilmpolicy"
    }
  }
}
'
# NB on my setup the ILM policy takes a LONG time to be applied, root cause unknown

# create security role
curl -k -u 'elastic:password' -X POST "http://localhost:9200/_security/role/role1?pretty" -H 'Content-Type: application/json' -d'
{
  "indices": [
    {
      "names": [ "twitter*", "partial-twitter*" ],
      "privileges": [ "read" ],
      "field_security" : {
        "grant" : [ "*" ],
        "except" : [ "category" ]
      }
    }
  ]
}
'

# create user
curl -k -u 'elastic:password' -X POST "http://localhost:9200/_security/user/jacknich?pretty" -H 'Content-Type: application/json' -d'
{
  "password" : "password",
  "roles" : [ "role1" ]
}
'

# search by the user
curl -k -u 'jacknich:password' -X GET "http://localhost:9200/*/_search?pretty"

@madisonb You would help us a lot if you could narrow down the reproduction a bit.
I think the index mapping and the role definition should suffice.

@ywelsch do you have any idea or hint on the issue on this?

[ywelsch]

@madisonb can you run the same failing command again with error_trace=true. This will provide more detailed information . Also can you share more details about the mappings / role configurations with @albertzaharovits so it's easier to reproduce? A diagnostics dump would help.

[madisonb]

@albertzaharovits did you push data up into the index and let it roll over into the frozen tier? My issue is when actual data rolls into the partial tier

Is there something specific you would like to see from the index mapping? The setup is using a component template and is roughly 800 lines long, and I would prefer not to post the entire thing as it exposes how our product is configured (for better or worse). I'm happy to provide a select/specific section if you'd like; we have dynamic templates, aliases, and traditional properties with all kinds of complex mappings that cover text, vectors, numbers, field data filters, geo, custom normalizers, etc.

@ywelsch Running the following command:

curl "https://user:pass@my-cluster.es.us-east-1.aws.found.io:9243/pulse/_count?error_trace=true"

doesn't seem to produce an error trace... Here's a snippet:

{"count":<partial number>,"_shards":{"total":309,"successful":14,"skipped":0,"failed":295,"failures":[{"shard":0,"index":"partial-pulse-000015","node":"XCRMYhdLR3KHuHxm74vlCg","reason":{"type":"unsupported_operation_exception","reason":"unsupported_operation_exception: null"}},
...
,{"shard":0,"index":"partial-pulse-000307","node":"XCRMYhdLR3KHuHxm74vlCg","reason":{"type":"unsupported_operation_exception","reason":"unsupported_operation_exception: null"}},{"shard":0,"index":"partial-pulse-000308","node":"9SNaA5L9TCqZ8l0BA39c1Q","reason":{"type":"unsupported_operation_exception","reason":"unsupported_operation_exception: null"}},{"shard":0,"index":"partial-pulse-000309","node":"XCRMYhdLR3KHuHxm74vlCg","reason":{"type":"unsupported_operation_exception","reason":"unsupported_operation_exception: null"}}]}}

Do I have that right?

Role definition that fails from my example above

GET /_security/role/pulse_data2

# returns
{
  "pulse_data2" : {
    "cluster" : [ ],
    "indices" : [
      {
        "names" : [
          "partial-*",
          "pulse",
          "pulse-*"
        ],
        "privileges" : [
          "read"
        ],
        "field_security" : {
          "grant" : [
            "*"
          ],
          "except" : [
            "type"
          ]
        },
        "allow_restricted_indices" : false
      }
    ],
    "applications" : [ ],
    "run_as" : [ ],
    "metadata" : { },
    "transient_metadata" : {
      "enabled" : true
    }
  }
}

Updating the role via Kibana to allow the _count query to return without failures is the following role (I've removed the field security except clause)

GET /_security/role/pulse_data2

# returns
{
  "pulse_data2" : {
    "cluster" : [ ],
    "indices" : [
      {
        "names" : [
          "partial-*",
          "pulse",
          "pulse-*"
        ],
        "privileges" : [
          "read"
        ],
        "field_security" : {
          "grant" : [
            "*"
          ],
          "except" : [ ]
        },
        "allow_restricted_indices" : false
      }
    ],
    "applications" : [ ],
    "run_as" : [ ],
    "metadata" : { },
    "transient_metadata" : {
      "enabled" : true
    }
  }
}

I can flip between these two role configurations to toggle the errors off and on for the user.

The other interesting thing as I am trying to get a better error trace for you is that I seem to be able to directly query the indices I care about that I know are frozen when the erroneous role is turned on without issue, but querying the partial indices don't work. Do I simply have my permissions incorrect and I shouldn't need the partial-* permission?

[ywelsch]

Do you get an error stack trace if you specify partial-pulse-000307 directly as index (instead of just pulse)?

[albertzaharovits]

@madisonb Thanks for the details so far. Alas, I'm still unable to reproduce.

did you push data up into the index and let it roll over into the frozen tier?

I have applied the policy manually, and my test policy triggers the searchable_snapshot action immediately.

I have created an alias (like 'pulse'), that points to both partially mounted indices and regular indices (indices that are in the frozen state, but not yet snapshotted and mounted, ie in the 'wait-for-shard-history-leases' and 'segment-count' steps), and querying the alias works fine for me.

I seem to be able to directly query the indices I care about that I know are frozen when the erroneous role is turned on without issue

I understand that the issue only manifests when querying the alias only? Can you confirm?

If we can't have the mapping, can you show the out of GET /_alias/pulse and GET _ilm/policy/<policy_id>.
Can you please double check that there are no stack exceptions in the node logs (this would be very odd)?

Are you able to contact Elastic support and mention this issue (I assume you'd be more comfortable sharing the information that way).

[madisonb]

@ywelsch

1. a direct query against partial-pulse-000307 works, no errors
2. a direct query against pulse-000307 works, no errors
3. a query against partial-pulse-* fails, and 000307 is included in the exception logs per my above comment
4. a query against the alias pulse fails just like prior
5. a query against pulse-* fails like prior

So perhaps I am having issues somewhere within the index pattern permissions or alias configuration?

@albertzaharovits

GET /_alias/pulse

# returns
{
  "partial-pulse-000145" : {
    "aliases" : {
      "pulse" : { }
    }
  },
  "partial-pulse-000169" : {
    "aliases" : {
      "pulse" : { }
    }
  },
  "partial-pulse-000097" : {
    "aliases" : {
      "pulse" : { }
    }
  },
.... (repeats for all indices)

ILM Policy

GET _ilm/policy/pulse


# see attached for return value

I can contact Elastic Support and reference this ticket as well.

ilm_policy.txt

[albertzaharovits]

a direct query against pulse-000307 works, no errors
a query against partial-pulse-* fails, and 000307 is included in the exception logs per my above comment

I take from this that the issue also manifests when the search ONLY includes searchable_snapshot shards. Moreover the same search request issued against selected searchable_snapshot shards encounters no such errors. In addition, the issue only happens if the user has FLS controls.

I haven't seen anything suspicious in the ILM policy.
The stuff that differs from my reproduction is that my cluster has a single node, and my index has one shard one segment. Plus the snapshot repository type, and ofc the index mapping.

This is an odd one.
I propose we continue the troubleshooting over the support ticket, but in the meantime can you also please run:

for index in $(curl -X GET "https://user:pass@my-cluster.es.us-east-1.aws.found.io:9243/_cat/indices/partial-pulse-*"); do
echo "try search $index"
curl -X GET "https://user:pass@my-cluster.es.us-east-1.aws.found.io:9243/$index/count"
done

, to verify if there's one single index/shard in a funny state, and which is then tripping the whole search request?

[madisonb]

So my specific user I've created does not have permission to _cat/indices, so I used another user to execute the _cat portion and then the regular user per above to do the _count

for index in $(curl -X GET "https://user2:pass2@my-cluster.es.us-east-1.aws.found.io:9243/_cat/indices/partial-pulse-*?h=index"); do
echo "try search $index"
curl -X GET "https://user:pass@my-cluster.es.us-east-1.aws.found.io:9243/$index/_count"
done

Returns

...
{"count":<>,"_shards":{"total":1,"successful":1,"skipped":0,"failed":0}}
try search partial-pulse-000210
{"count":<>,"_shards":{"total":1,"successful":1,"skipped":0,"failed":0}}
try search partial-pulse-000219
{"count":<>,"_shards":{"total":1,"successful":1,"skipped":0,"failed":0}}
try search partial-pulse-000217
{"count":<>,"_shards":{"total":1,"successful":1,"skipped":0,"failed":0}}
try search partial-pulse-000218
{"count":2698412,"_shards":{"total":1,"successful":1,"skipped":0,"failed":0}}
...

All indices were successful, no errors.