Missing error metrics on new_relic sink

[bosouza]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

I was trying to use Vector sink metrics to determine a "sink health", but when looking at the metrics emitted by the new_relic sink when configured with incorrect credentials I didn't find component_errors_total or component_discarded_events_total.

Also, component_received_events_total and component_sent_events_total both increase together even though none of the logs actually reach the provider because the credentials are wrong.

There are also some metrics which are not documented in the telemetry section for the sink: http_client_requests_sent_total and http_client_responses_total.

Please run the config provided in this report to reproduce the behavior I'm seeing.

Configuration

{
    "sources": {
        "demo-logs": {
            "type": "demo_logs",
            "format": "json"
        },
        "internal-metrics": {
            "type": "internal_metrics"
        }
    },
    "transforms": {
        "internal-metrics-as-logs": {
            "type": "metric_to_log",
            "inputs": ["internal-metrics"]
        }
    },
    "sinks": {
        "new-relic": {
            "type": "new_relic",
            "inputs": ["demo-logs"],
            "account_id": "10101010",
            "license_key": "invalidcredentials",
            "api": "logs"
        },
        "sink-metrics": {
            "type": "file",
            "encoding": {"codec": "json"},
            "path": "./new-relic-sink-metrics.log",
            "inputs": ["internal-metrics-as-logs"]
        }
    }
}

Version

vector 0.32.0 (x86_64-unknown-linux-gnu 1b403e1 2023-08-15 14:56:36.089460954)

Debug Output

No response

Example Data

No response

Additional Context

No response

References

No response

[pront]

Similar to #18296.

[bosouza]

@pront this is a bit different

Also, component_received_events_total and component_sent_events_total both increase together even though none of the logs actually reach the provider because the credentials are wrong.

if the events are stuck forever on retries like in #18296 then they shouldn't be counting towards component_sent_events_total right?

[dsmith3197]

Hey @bosouza,

What status codes do you see reported for http_client_responses_total?

[bosouza]

looking at the logs from my tests a few months ago it's all 403s. Just to be sure I ran the example config from the issue description with vector 0.34.1 and got the same thing:

{"counter":{"value":1.0},"host":"m15","kind":"absolute","name":"http_client_responses_total","namespace":"vector","tags":{"status":"403"},"timestamp":"2023-12-18T19:11:09.052219083Z"}

[dsmith3197]

@bosouza Thanks, I think I see the problem. I took a quick look at the source code for the new relic sink and we actually ignore the status code of the response and always report a successful delivery, which explains the behavior you are experiencing.

vector/src/sinks/new_relic/service.rs

Line 102 in 5ef73ee

Ok(_) => Ok(NewRelicApiResponse {