multiple update of one document in one _bulk too slow

[dizzzyroma]

Hi,

Multiple update of one document in one _bulk is too slow

Example:

curl -XPOST "http://localhost:9200/_bulk" -H 'Content-Type: application/json' -d'
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":1}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":2}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":1}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":2}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":1}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":2}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":1}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":2}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":1}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":2}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":1}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":2}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":1}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":2}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":1}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":2}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":1}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":2}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":1}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":2}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":1}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":2}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":1}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":2}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":1}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":2}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":1}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":2}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":1}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":2}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":1}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":2}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":1}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":2}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":1}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":2}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":1}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":2}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":1}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":2}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":1}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":2}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":1}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":2}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":1}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":2}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":1}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":2}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":1}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":2}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":1}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":2}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":1}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":2}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":1}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":2}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":1}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":2}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":1}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":2}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":1}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":2}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":1}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":2}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":1}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":2}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":1}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":2}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":1}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":2}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":1}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":2}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":1}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":2}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":1}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":2}}
'

Elasticsearch version: 5.3, 5.2

Plugins installed: []

JVM version: 1.8.0_121-b13

OS version:Ubuntu 16.04.2 LTS

bulk request with 75 updates takes near 6 seconds on our test index that runs on c3.xlarge (4 CPU, 8 GB, documents in inex 14860619, index size 20GB) .
bulk request with 150 updates takes near 12 seconds.

bulk index with 75 documents takes 0.2 seconds

On ES 2.4 - we haven't this problem

Profiler info:

If it helps, I may open profiling port to cluster where this problem is reproduced

[jimczi]

In 5.x the global ordinals are built during the refresh:
https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-parent-field.html#_global_ordinals
This is a trade off to make sure that the first query with parent/child on the new searcher can run fast. You can disable this behavior as explained in the documentation or you can disable refresh when you're bulk updating your index.
Anyhow this discussion is better suited for the forum. I am going to close this issue because we reserve github for verified bugs and new features. If you have any questions we can continue the discussion in the forum:
https://discuss.elastic.co/c/elasticsearch

[jimczi]

My previous comment is not entirely true. The refresh is triggered by the updates itself so disabling the refresh would not solve your problem. In 5.x we rely on refresh to update a document that is not flushed yet:
#20102
This change coupled with the fact that we eagerly load the parent/child ordinals on refresh makes your ingestion rate slow. I'd suggest to revisit your ingestion method by maybe grouping updates of a single document in one when you can ?

[dizzzyroma]

Thanks for answer I create thread for this https://discuss.elastic.co/t/multiple-update-of-one-document-in-one--bulk-too-slow/82885
I research it