Collector internal telemetry updates #4867

danelson · 2024-07-23T16:56:21Z

~~Internal collector detailed metrics were missing the otelcol_ prefix.~~
log_records were missing from the critical monitoring sections (I assume this is not by design)

danelson · 2024-07-23T20:45:32Z

I just came across open-telemetry/opentelemetry-collector#9315 and open-telemetry/opentelemetry-collector#9759 so now I am not sure this is correct. Perhaps it is because I am consuming the metrics on localhost:8888/metrics. Anyway, hopefully a maintainer can clarify.

codeboten

Thanks @danelson, it's a bit confusing, but these metrics listed in the table will not be prefixed because they're emitted by instrumentation libraries rather than by collector components

codeboten · 2024-07-24T15:33:21Z

content/en/docs/collector/internal-telemetry.md

+| `otelcol_http_client_active_requests`     | Number of active HTTP client requests.                                                    | Counter   |
+| `otelcol_http_client_connection_duration` | Measures the duration of the successfully established outbound HTTP connections.          | Histogram |
+| `otelcol_http_client_open_connections`    | Number of outbound HTTP connections that are active or idle on the client.                | Counter   |
+| `otelcol_http_client_request_body_size`   | Measures the size of HTTP client request bodies.                                          | Histogram |
+| `otelcol_http_client_request_duration`    | Measures the duration of HTTP client requests.                                            | Histogram |
+| `otelcol_http_client_response_body_size`  | Measures the size of HTTP client response bodies.                                         | Histogram |
+| `otelcol_http_server_active_requests`     | Number of active HTTP server requests.                                                    | Counter   |
+| `otelcol_http_server_request_body_size`   | Measures the size of HTTP server request bodies.                                          | Histogram |
+| `otelcol_http_server_request_duration`    | Measures the duration of HTTP server requests.                                            | Histogram |
+| `otelcol_http_server_response_body_size`  | Measures the size of HTTP server response bodies.                                         | Histogram |
+| `otelcol_rpc_client_duration`             | Measures the duration of outbound RPC.                                                    | Histogram |
+| `otelcol_rpc_client_request_size`         | Measures the size of RPC request messages (uncompressed).                                 | Histogram |
+| `otelcol_rpc_client_requests_per_rpc`     | Measures the number of messages received per RPC. Should be 1 for all non-streaming RPCs. | Histogram |
+| `otelcol_rpc_client_response_size`        | Measures the size of RPC response messages (uncompressed).                                | Histogram |
+| `otelcol_rpc_client_responses_per_rpc`    | Measures the number of messages sent per RPC. Should be 1 for all non-streaming RPCs.     | Histogram |
+| `otelcol_rpc_server_duration`             | Measures the duration of inbound RPC.                                                     | Histogram |
+| `otelcol_rpc_server_request_size`         | Measures the size of RPC request messages (uncompressed).                                 | Histogram |
+| `otelcol_rpc_server_requests_per_rpc`     | Measures the number of messages received per RPC. Should be 1 for all non-streaming RPCs. | Histogram |
+| `otelcol_rpc_server_response_size`        | Measures the size of RPC response messages (uncompressed).                                | Histogram |
+| `otelcol_rpc_server_responses_per_rpc`    | Measures the number of messages sent per RPC. Should be 1 for all non-streaming RPCs.     | Histogram |


These metrics are generated by the underlying instrumentation library, not by collector components. This means that the prefix will not be present here

@codeboten are you saying to revert this? Also can you clarify how this affects consuming the data when scraping the metrics with the prometheus receiver? I am running 0.104.0 with the below config and I see metrics with names like otelcol_http_server_response_size (note it is not response_body_size either)

OTel config

receivers: otlp: protocols: http: endpoint: 0.0.0.0:4318 prometheus/collector: config: scrape_configs: - job_name: "internal" scrape_interval: 10s static_configs: - targets: - "localhost:8888" processors: filter/collector: error_mode: ignore metrics: include: match_type: regexp metric_names: - .*http_server.* exporters: debug: verbosity: detailed service: telemetry: metrics: level: detailed pipelines: logs: receivers: [otlp] exporters: [debug] metrics: receivers: [prometheus/collector] processors: [filter/collector] exporters: [debug]

Right, until v0.106.1, the prometheus exporter configured inside the collector was setting a prometheus namespace of otelcol to prefix all metrics exported via prometheus. This was inconsistent with the metrics exported via other exporters (OTLP, console). This was addressed by prefixing all collector component generated metrics manually with otelcol_ to provide a consistent metric name for all exporters. Note that when I'm using the term exporters here, I mean the exporters configured inside the Collector for the use of the OTel Go SDK.

This means that all metrics generated by instrumentation libraries will match the names that these instrumentation libraries intended as per the example below, where http_server_response_size used to be prefixed by otelcol_ and will now look like this:

http_server_response_size{http_method="POST",http_scheme="http",http_status_code="200",net_host_name="127.0.0.1",net_host_port="4318",net_protocol_name="http",net_protocol_version="1.1",service_instance_id="aa3d8988-fdf1-4023-8fff-193877983817",service_name="otelcontribcol",service_version="0.106.1-dev"} 2

Thank you. The disconnect I was having was between collector generated and instrumentation library generated metrics.

See Alex's comment

tiffany76

Two small changes, along with Alex's note. Thanks!

content/en/docs/collector/internal-telemetry.md

svrnm · 2024-08-01T09:18:56Z

@danelson please take a look at the requested changes, thanks!

Co-authored-by: Tiffany Hrabusa <30397949+tiffany76@users.noreply.github.com>

danelson · 2024-08-01T20:48:21Z

I think this should be good now. Thank you for the feedback.

tiffany76 · 2024-08-01T23:05:07Z

Thanks for the changes, @danelson!

Also, I created #4933 to make sure we clarify the change in naming conventions with v0.106.1.

cartermp · 2024-08-03T18:21:19Z

@codeboten PTAL!

codeboten

Thanks @danelson!

theletterf · 2024-09-05T19:00:22Z

/fix:format

opentelemetrybot · 2024-09-05T19:00:36Z

You triggered fix:format action run at https://github.com/open-telemetry/opentelemetry.io/actions/runs/10726531500

opentelemetrybot · 2024-09-05T19:00:57Z

fix:format was successful.

IMPORTANT: (RE-)RUN `/fix:all` to ensure that there are no remaining check issues.

tiffany76 · 2024-09-05T19:34:10Z

/fix:all

opentelemetrybot · 2024-09-05T19:34:23Z

You triggered fix:all action run at https://github.com/open-telemetry/opentelemetry.io/actions/runs/10726991011

opentelemetrybot · 2024-09-05T19:38:34Z

fix:all was successful.

IMPORTANT: (RE-)RUN `/fix:all` to ensure that there are no remaining check issues.

Co-authored-by: Tiffany Hrabusa <30397949+tiffany76@users.noreply.github.com> Co-authored-by: Alex Boten <223565+codeboten@users.noreply.github.com> Co-authored-by: opentelemetrybot <107717825+opentelemetrybot@users.noreply.github.com> Co-authored-by: Phillip Carter <pcarter@fastmail.com>

danelson added 2 commits July 23, 2024 11:52

Update names of collector detailed metrics

a308a89

Include log_records in critical monitoring

c0b09bc

danelson requested a review from a team July 23, 2024 16:56

github-actions bot added the sig:collector label Jul 23, 2024

svrnm requested review from a team and mx-psi and removed request for a team July 23, 2024 17:19

mx-psi requested a review from codeboten July 24, 2024 08:40

mx-psi previously approved these changes Jul 24, 2024

View reviewed changes

codeboten reviewed Jul 24, 2024

View reviewed changes

tiffany76 requested changes Jul 24, 2024

View reviewed changes

content/en/docs/collector/internal-telemetry.md Outdated Show resolved Hide resolved

content/en/docs/collector/internal-telemetry.md Outdated Show resolved Hide resolved

Oxford comma

4bd946c

Co-authored-by: Tiffany Hrabusa <30397949+tiffany76@users.noreply.github.com>

opentelemetrybot requested review from a team and djaglowski and removed request for a team August 1, 2024 15:47

Oxford comma

bab9b27

Co-authored-by: Tiffany Hrabusa <30397949+tiffany76@users.noreply.github.com>

opentelemetrybot requested a review from a team August 1, 2024 15:48

Revert prefix for instrumentation library metrics

92eff9c

danelson requested review from codeboten and tiffany76 August 1, 2024 20:48

tiffany76 approved these changes Aug 1, 2024

View reviewed changes

tiffany76 mentioned this pull request Aug 1, 2024

Clarify Collector internal metric names by version #4933

Closed

cartermp added the sig-approval-missing Co-owning SIG didn't provide an approval label Aug 3, 2024

codeboten approved these changes Sep 5, 2024

View reviewed changes

Merge branch 'main' into internal-telemetry-updates

b55a326

opentelemetrybot requested a review from a team September 5, 2024 18:54

Results from /fix:format

25316e8

Merge branch 'main' into internal-telemetry-updates

11cf5ac

cartermp approved these changes Sep 7, 2024

View reviewed changes

cartermp added this pull request to the merge queue Sep 7, 2024

Merged via the queue into open-telemetry:main with commit e4f6838 Sep 7, 2024
17 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Collector internal telemetry updates #4867

Collector internal telemetry updates #4867

danelson commented Jul 23, 2024 •

edited

Loading

danelson commented Jul 23, 2024 •

edited

Loading

codeboten left a comment

codeboten Jul 24, 2024

danelson Aug 1, 2024 •

edited

Loading

codeboten Aug 1, 2024

danelson Aug 1, 2024

tiffany76 left a comment

svrnm commented Aug 1, 2024

danelson commented Aug 1, 2024

tiffany76 commented Aug 1, 2024

cartermp commented Aug 3, 2024

codeboten left a comment

theletterf commented Sep 5, 2024

opentelemetrybot commented Sep 5, 2024

opentelemetrybot commented Sep 5, 2024

tiffany76 commented Sep 5, 2024

opentelemetrybot commented Sep 5, 2024

opentelemetrybot commented Sep 5, 2024

Collector internal telemetry updates #4867

Collector internal telemetry updates #4867

Conversation

danelson commented Jul 23, 2024 • edited Loading

danelson commented Jul 23, 2024 • edited Loading

codeboten left a comment

Choose a reason for hiding this comment

codeboten Jul 24, 2024

Choose a reason for hiding this comment

danelson Aug 1, 2024 • edited Loading

Choose a reason for hiding this comment

codeboten Aug 1, 2024

Choose a reason for hiding this comment

danelson Aug 1, 2024

Choose a reason for hiding this comment

tiffany76 left a comment

Choose a reason for hiding this comment

svrnm commented Aug 1, 2024

danelson commented Aug 1, 2024

tiffany76 commented Aug 1, 2024

cartermp commented Aug 3, 2024

codeboten left a comment

Choose a reason for hiding this comment

theletterf commented Sep 5, 2024

opentelemetrybot commented Sep 5, 2024

opentelemetrybot commented Sep 5, 2024

IMPORTANT: (RE-)RUN /fix:all to ensure that there are no remaining check issues.

tiffany76 commented Sep 5, 2024

opentelemetrybot commented Sep 5, 2024

opentelemetrybot commented Sep 5, 2024

IMPORTANT: (RE-)RUN /fix:all to ensure that there are no remaining check issues.

danelson commented Jul 23, 2024 •

edited

Loading

danelson commented Jul 23, 2024 •

edited

Loading

danelson Aug 1, 2024 •

edited

Loading

IMPORTANT: (RE-)RUN `/fix:all` to ensure that there are no remaining check issues.

IMPORTANT: (RE-)RUN `/fix:all` to ensure that there are no remaining check issues.