Trying to set a password in ES 8.0 fails for a security index that was created in 7.0

[joshdover]

Describe the feature:

Elasticsearch version (bin/elasticsearch --version): 8.0.0-SNAPSHOT

Plugins installed: []

JVM version (java -version): openjdk version "11.0.2" 2019-01-15

OS version (uname -a if on a Unix-like system): Darwin Kernel Version 18.2.0

Description of the problem including expected versus actual behavior:

I should be able to update passwords for users created in 7.0 on a 8.0 cluster.

Steps to reproduce:

Please include a minimal but complete recreation of the problem, including
(e.g.) index creation, mappings, settings, query etc. The easier you make for
us to reproduce it, the more likely that somebody will take the time to look at it.

1. Start a 7.0 node with Security enabled, create a user
2. Shutdown node
3. Start an ES 8.0 node with data from the 7.0 node (also with Security enabled)
4. Try to update a user's password using the PUT /_security/user/<username>/_password API
5. Get this response:

{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Rejecting mapping update to [.reindexed-v7-security-6] as the final mapping would have more than 1 type: [_doc, doc]"}],"type":"illegal_argument_exception","reason":"Rejecting mapping update to [.reindexed-v7-security-6] as the final mapping would have more than 1 type: [_doc, doc]"},"status":400}

When looking at the mappings for the .security index it appears to only have a single type (_doc). My guess is that the logic for putting the mappings internally in ES is using doc rather than just no type at all.

Provide logs (if relevant):

09:55:12      │ info [o.e.a.a.i.m.p.TransportPutMappingAction] [kibana-ci-immutable-ubuntu-1549898516931060281] failed to put mappings on indices [[[.reindexed-v7-security-6/MHu7LjQqRB2eD4DkS2b_xg]]], type [doc]
09:55:12      │      java.lang.IllegalArgumentException: Rejecting mapping update to [.reindexed-v7-security-6] as the final mapping would have more than 1 type: [_doc, doc]
09:55:12      │      	at org.elasticsearch.index.mapper.MapperService.internalMerge(MapperService.java:449) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:55:12      │      	at org.elasticsearch.index.mapper.MapperService.internalMerge(MapperService.java:398) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:55:12      │      	at org.elasticsearch.index.mapper.MapperService.merge(MapperService.java:331) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:55:12      │      	at org.elasticsearch.cluster.metadata.MetaDataMappingService$PutMappingExecutor.applyRequest(MetaDataMappingService.java:315) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:55:12      │      	at org.elasticsearch.cluster.metadata.MetaDataMappingService$PutMappingExecutor.execute(MetaDataMappingService.java:238) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:55:12      │      	at org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:687) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:55:12      │      	at org.elasticsearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:310) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:55:12      │      	at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:210) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:55:12      │      	at org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:142) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:55:12      │      	at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:55:12      │      	at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:55:12      │      	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:681) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:55:12      │      	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:252) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:55:12      │      	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:215) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:55:12      │      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
09:55:12      │      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
09:55:12      │      	at java.lang.Thread.run(Thread.java:834) [?:?]

[elasticmachine]

Pinging @elastic/es-security

[tvernum]

I can't reproduce this using the standard snapshots. I suspect it is something to do with the builds that Kibana is running (there may still be a real bug, it's just that your listed reproduction steps aren't enough to trigger it).

I don't understand where this index name comes from: .reindexed-v7-security-6
That's not the name of the default security index, and I'm not aware of anything that creates that index. Do you know why your CI has it?

The security code uses doc for its type. If your .security index has _doc, then something strange has happened.

My steps were

1. downloaded the latest snapshots of 7.0.0-beta1 and 8.0.0-SNAPSHOT (details below)
2. Set the 7.0.0 config to

   node.name: node1
   cluster.name: es7-8-test
   xpack.license.self_generated.type: trial
   xpack.security.enabled: true
   xpack.watcher.enabled: false
   xpack.monitoring.enabled: false
   

3. set the bootstrap password for 7.0.0
4. started ES 7.0.0
5. Created a user

   PUT /_security/user/foo
   {
     "password": "magic123", "roles": []
   }
   

6. Shutdown that node
7. Copied the 7.0.0 config and the data directory to the 8.0.0 directory
8. Set the bootstrap password for 8.0.0
9. Started 8.0.0
10. Change the user's password

PUT /_security/user/foo/_password
{
  "password": "hunter"
}

ES 7.0.0 build details

• commit https://github.com/elastic/elasticsearch/commits/9fae0a4
• build_id 7.0.0-beta1-715764f6
• build start_time Mon, 11 Feb 2019 18:08:19 GMT
• package elasticsearch-7.0.0-beta1-darwin-x86_64.tar.gz

ES 8.0.0 build details

• commit https://github.com/elastic/elasticsearch/commits/4e9b1cf
• build_id 8.0.0-fcd45d56
• build start_time "Sat, 9 Feb 2019 07:39:22 GMT
• package elasticsearch-8.0.0-SNAPSHOT-darwin-x86_64.tar.gz

[joshdover]

Great, this is helpful. I think I have found the issue.

The security code uses doc for its type. If your .security index has _doc, then something strange has happened.

The mapping with type _doc gets created when using the Upgrade Assistant in 8.0 to reindex 7.x indices in preparation for 9.0. Because the Upgrade Assistant uses the typeless API in 7.0+ (which I believe is the only way to create new indices), I'm guessing Elasticsearch creates a mapping with _doc under the hood. Seems to me the internal mappings for .security need to be updated to use _doc or no type at all.

The only change needed in your steps to reproduce the issue would be between steps (9) and (10). Before Step 10:

• Startup Kibana 8.0 against ES 8.0
• Start the reindex. Currently this has to be done via API b/c the Elasticsearch Deprecations API does not mark any indices in 8.0 as need to be reindexed for the next Lucene version, so the UI does not show them.
  • curl -u elastic:changeme -XPOST localhost:5601/api/upgrade_assistant/reindex/.security-6 -H 'kbn-xsrf: true'
• Wait for the reindex to complete. You'll need to wait until the reindexOp.status field of this response is equal to 1:
  • curl -u elastic:changeme -XGET localhost:5601/api/upgrade_assistant/reindex/.security-6
• Look at the new mappings for .security, verify you see _doc now instead of doc.
  • curl "localhost:9200/.security/_mappings?include_type_name=true" -u elastic:changeme

[tvernum]

@joshdover
Thanks for your investigation. I'd like to close this issue and open a new one about the security index not being compatible with the Kibana upgrade assistant for 7.x to 8.x

It's not particularly surprising that this doesn't work - we haven't done any work in master to support security upgrades from 7 to 8.

Per #38678 (comment), we very much hope that security will stop using a regular index by the time we get to 8, so we're not likely to prioritise work that would change the type used in the security inde (or move to type-less APIs) until we have a better feel about where we're going with "system indices" in 7.x/8.0

For now can you just not run the upgrade assistant on the security index?

[Blackiie]

Emm ok no entiendo muy bien pero trataré de comprenderlo 😀. Gracias por la atencion!! El 12 feb. 2019 19:47, "Tim Vernum" <notifications@github.com> escribió:

…

@joshdover <https://github.com/joshdover> Thanks for your investigation. I'd like to close this issue and open a new one about the security index not being compatible with the Kibana upgrade assistant for 7.x to 8.x It's not particularly surprising that this doesn't work - we haven't done any work in master to support security upgrades from 7 to 8. Per #38678 <#38678>, we very much hope that security will stop using a regular index by the time we get to 8, so we're not likely to prioritise work that would change the type used in the security inde (or move to type-less APIs) until we have a better feel about where we're going with "system indices" in 7.x/8.0 For now can you just not run the upgrade assistant on the security index? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#38740 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AtTlEwea3ZAIrR_Nr0uCgUmQM75vzfvWks5vM4rbgaJpZM4a0ig9> .

[joshdover]

@tvernum That works for me. Just wanted to flag this early so it wasn't forgotten about. For our testing purposes, we'll exclude .security for the time being. Thanks!