OTLPHTTP exporter ignores response code and content-type, retries and logs anyway

[breedx-splk]

There exists a regression/bug in 0.81.0 (and 82 and 83) of the otel-collector-contrib
with the otlphttp exporter. This bug is initially observed when debug logging for
trace export is enabled:

{"kind": "exporter", "data_type": "traces", "name": "logging/debug"}
2023-08-22T15:36:59.784-0700	info	exporterhelper/queued_retry.go:423	Exporting failed. Will retry the request after interval.	{"kind": "exporter", "data_type": "traces", "name": "otlphttp", "error": "unexpected EOF", "interval": "7.214643193s"}

The collector has sent a successful otlphttp payload to a backend and received an HTTP 200
success response, but fails to parse the return body and thus unnecessarily re-queues
the data for retransmission.

The result is both a confusing error message to the user and a duplicate sending
of payload data to the backend. This wastes resources and could potentially cause
other larger problems for backends.

It is believed that the root cause was introduced when the partial success
behavior was added in 0.81.0. This assumes that a response payload from the backend
always contains protobuf, which is not necessarily true. In this repro case,
the response is clearly json.

Steps to reproduce

See the details in this repro repo: https://github.com/breedx-splk/collector_otlphttp_eof

What did you expect to see?

Successful exports should not log an error and definitely should not retry, regardless of the content-type of the response.

What did you see instead?

An error and the collector retries, even tho data was exported correctly.

What version did you use?

Verified as problematic on 0.81.0, 0.82.0, and 0.83.0

What config did you use?

See above repro repo for config.

Environment

Tested on MacOS.

Additional context

https://github.com/breedx-splk/collector_otlphttp_eof

[breedx-splk]

This PR #6970 seems to ignore the response Content-type and simply assume protobuf response regardless. When this happens, other responses (text, json, whatever) are unparseable and the collector logs an error (at debug at least) and queues the data for retry.

[tigrannajaryan]

@atoulme is it possible to also improve the error message when the response is really not parseable?

[atoulme]

@atoulme is it possible to also improve the error message when the response is really not parseable?

Please see https://github.com/open-telemetry/opentelemetry-collector/pull/8283/files thanks!