The entire bulk request fails if any document parsing fails with pipeline configured

[samxbr]

Elasticsearch Version

8.18, 8.19, 9.0, 9.1, 9.2 (and older versions that are no longer maintained)

Problem Description

Normally in a bulk request, if indexing for individual document fails (eg. due to invalid json), the rest of the documents in the bulk request will still succeed, and respond a 200 status, with error details for individual doc in the body. However, if the failing document has a pipeline configured, the entire request will fail with 400 status and none of the documents will be indexed.

This is reported by https://github.com/elastic/sdh-elasticsearch/issues/9486, and this comment pointed we may need to tighten up the error handling in IngestService.

Steps to Reproduce

Create a dummy pipeline

PUT /_ingest/pipeline/test-ds-pl-1
{
  "processors": [
    {
      "append": {
        "field": "name",
        "value": "value"
      }
    }
  ]
}

Bulk index invalid json doc entry without pipeline

PUT /_bulk
{"create":{"_index":"test-ds-1"}}
{"@timestamp":"2024-06-01T00:00:00Z"}
{"create":{"_index":"test-ds-1"}}
{"@timestamp":"2024-06-01T00:00:00Z","@timestamp":"2024-07-01T00:00:00Z"}

• Responds 200, the first doc is indexed, the second doc is not. This is expected.

Bulk index invalid json doc entry with a pipeline

• note the duplicated @timestamp field

PUT /_bulk?pipeline=test-ds-pl-1
{"create":{"_index":"test-ds-1"}}
{"@timestamp":"2024-06-01T00:00:00Z"}
{"create":{"_index":"test-ds-1"}}
{"@timestamp":"2024-06-01T00:00:00Z","@timestamp":"2024-07-01T00:00:00Z"}

• Responds 400 and no docs are indexed.

Bulk index invalid json doc entry with a pipeline

• note the missing closing bracket in the second doc

PUT /_bulk?pipeline=test-ds-pl-1
{"create":{"_index":"test-ds-1"}}
{"@timestamp":"2024-06-01T00:00:00Z"}
{"create":{"_index":"test-ds-1"}}
{"@timestamp":"2024-06-01T00:00:00Z"

• Responds 400 and no docs are indexed.

[elasticsearchmachine]

Pinging @elastic/es-data-management (Team:Data Management)

[samxbr]

The team needs to discuss how invalid JSON error interacts with failure store. Due to potential ramification of handling invalid documents in failure store, failure store will not be considered as part of fixing this issue.

[samxbr]

The bug fix will be backported to the currently maintained versions 8.19, 9.1 and 9.2.