component_discarded_events_total incorrect value on elasticsearch sink

[pjarmalavicius]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

Elasticsearch sink exposes incorrect component_discarded_events_total metric value. Logs also displays incorrect count value.

It seems that metric is incremented by the total events in requests value, e.g. if request has 3 events and 2 of them fails, metric is incremented by 3 (log message also shows 3). This means that it's not possible to know exact number of lost events.

Actually elasticsearch response returns failed items very nicely and valid failed items count be parsed quite easily. Here is example elasticsearch response:

{
  "took": 6,
  "errors": true,
  "items": [
    {
      "index": {
        "_index": "logging-test-2023.12.06",
        "_type": "_doc",
        "_id": "111",
        "_version": 1,
        "result": "created",
        "_shards": {
          "total": 2,
          "successful": 1,
          "failed": 0
        },
        "_seq_no": 12,
        "_primary_term": 1,
        "status": 201
      }
    },
    {
      "index": {
        "_index": "logging-test-2023.12.06",
        "_type": "_doc",
        "_id": "222",
        "status": 400,
        "error": {
          "type": "mapper_parsing_exception",
          "reason": "***",
          "caused_by": {
            "type": "illegal_state_exception",
            "reason": "Can't get text on a START_OBJECT at 1:54"
          }
        }
      }
    },
    {
      "index": {
        "_index": "logging-test-2023.12.06",
        "_type": "_doc",
        "_id": "333",
        "status": 400,
        "error": {
          "type": "mapper_parsing_exception",
          "reason": "***",
          "caused_by": {
            "type": "illegal_state_exception",
            "reason": "Can't get text on a START_OBJECT at 1:54"
          }
        }
      }
    }
  ]
}

Configuration

No response

Version

0.34.1

Debug Output

No response

Example Data

{"level": "info","msg": "test-1", "foo": "some-string"}
{"level": "info","msg": "test-2", "foo": {"aa": "bb"}}
{"level": "info","msg": "test-3", "foo": {"aa": "cc"}}

Additional Context

No response

References

No response

[jszwedko]

This is unfortunately somewhat of a known issue. The elasticsearch sink doesn't handle partial failures particularly well:

• it doesn't increment the counts for just failed events
• when it retries failed requests, it resends all events regardless of whether they failed (Elasticsearch partial bulk failures #140)

[pjarmalavicius]

This is unfortunately somewhat of a known issue. The elasticsearch sink doesn't handle partial failures particularly well:

• it doesn't increment the counts for just failed events
• when it retries failed requests, it resends all events regardless of whether they failed (Elasticsearch partial bulk failures #140)

Are there any plans to work on it?

[jszwedko]

This is unfortunately somewhat of a known issue. The elasticsearch sink doesn't handle partial failures particularly well:

• it doesn't increment the counts for just failed events
• when it retries failed requests, it resends all events regardless of whether they failed (Elasticsearch partial bulk failures #140)

Are there any plans to work on it?

It's not on our immediate roadmap, but we would be happy to see PRs in this area 🙏