Failure Store: Update go-docappender to respect failure store status

[1pkg]

Description

This PR updates go-docappender to respect new failure store response status and emit new correspodning metrics accordingly. The old "indexed" metrics stay intact instead a new separate set of failure store labels is exposed.

This PR depends on changes made for BulkIndexerResponseItem in elastic/go-elasticsearch#948 and should only be merged afterwards.

How to test:

Use an instance of ES that has failure store feature enabled and enable failure store for a data stream via component template with.

POST _component_template/$name
{
  "template": {
    "data_stream_options": {
      "failure_store": {
        "enabled": true
      }
    }
  }
}

Set a custom "fail" ingest pipeline with.

PUT _ingest/pipeline/$name
{
  "processors": [
    {
      "fail": {
        "message": "fail"
      }
    }
  ]
}

Ingest some data to corresponding data stream, then check that failure store metrics are getting reported correctly.

To emulate "failed" status, set backing data stream index to be read only with.

PUT /$index/_settings
{
  "index.blocks.read_only": true
}

[axw]

Looks good! Just one question

[axw]

One more question, sorry: should we be measuring "not_enabled" too?

[1pkg]

One more question, sorry: should we be measuring "not_enabled" too?

Agree, this metric could be useful too. Updated to expose it in the last commit.

[marclop]

I think this looks good overall, just one small question

[marclop]

LGTM!

The raised concerns were addressed.