Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[translator/loki] Convert dots to underscores in labels names #19373

Merged

Conversation

mar4uk
Copy link
Contributor

@mar4uk mar4uk commented Mar 7, 2023

Description:
Loki doesn't support label names containing dots.
Added label names normalization to follow Prometheus label names standard. Dots in label names will be converted to underscores before sending them to Loki

Link to tracking Issue: #14113

@runforesight
Copy link

runforesight bot commented Mar 7, 2023

Foresight Summary

    
Major Impacts

build-and-test-windows duration(5 seconds) has decreased 36 minutes 40 seconds compared to main branch avg(36 minutes 45 seconds).
View More Details

⭕  build-and-test-windows workflow has finished in 5 seconds (36 minutes 50 seconds less than main branch avg.) and finished at 22nd Mar, 2023.


Job Failed Steps Tests
windows-unittest-matrix -     🔗  N/A See Details
windows-unittest -     🔗  N/A See Details

✅  telemetrygen workflow has finished in 57 seconds (43 seconds less than main branch avg.) and finished at 22nd Mar, 2023.


Job Failed Steps Tests
build-dev -     🔗  N/A See Details
publish-latest -     🔗  N/A See Details
publish-stable -     🔗  N/A See Details

✅  check-links workflow has finished in 1 minute 47 seconds (⚠️ 40 seconds more than main branch avg.) and finished at 22nd Mar, 2023.


Job Failed Steps Tests
changed files -     🔗  N/A See Details
check-links -     🔗  N/A See Details

✅  changelog workflow has finished in 2 minutes 11 seconds and finished at 22nd Mar, 2023.


Job Failed Steps Tests
changelog -     🔗  N/A See Details

✅  build-and-test workflow has finished in 40 minutes 58 seconds (11 minutes 25 seconds less than main branch avg.) and finished at 22nd Mar, 2023.


Job Failed Steps Tests
unittest-matrix (1.19, connector) -     🔗  ✅ 126  ❌ 0  ⏭ 0    🔗 See Details
unittest-matrix (1.20, connector) -     🔗  ✅ 126  ❌ 0  ⏭ 0    🔗 See Details
unittest-matrix (1.19, internal) -     🔗  ✅ 583  ❌ 0  ⏭ 0    🔗 See Details
correctness-traces -     🔗  ✅ 17  ❌ 0  ⏭ 0    🔗 See Details
unittest-matrix (1.20, internal) -     🔗  ✅ 583  ❌ 0  ⏭ 0    🔗 See Details
correctness-metrics -     🔗  ✅ 2  ❌ 0  ⏭ 0    🔗 See Details
unittest-matrix (1.19, extension) -     🔗  ✅ 543  ❌ 0  ⏭ 0    🔗 See Details
unittest-matrix (1.19, processor) -     🔗  ✅ 1557  ❌ 0  ⏭ 0    🔗 See Details
unittest-matrix (1.20, extension) -     🔗  ✅ 543  ❌ 0  ⏭ 0    🔗 See Details
unittest-matrix (1.20, processor) -     🔗  ✅ 1557  ❌ 0  ⏭ 0    🔗 See Details
unittest-matrix (1.19, receiver-0) -     🔗  ✅ 2611  ❌ 0  ⏭ 0    🔗 See Details
unittest-matrix (1.20, receiver-0) -     🔗  ✅ 2611  ❌ 0  ⏭ 0    🔗 See Details
unittest-matrix (1.19, exporter) -     🔗  ✅ 2501  ❌ 0  ⏭ 0    🔗 See Details
unittest-matrix (1.20, exporter) -     🔗  ✅ 2501  ❌ 0  ⏭ 0    🔗 See Details
unittest-matrix (1.20, receiver-1) -     🔗  ✅ 1969  ❌ 0  ⏭ 0    🔗 See Details
unittest-matrix (1.19, receiver-1) -     🔗  ✅ 1969  ❌ 0  ⏭ 0    🔗 See Details
unittest-matrix (1.20, other) -     🔗  ✅ 4748  ❌ 0  ⏭ 0    🔗 See Details
unittest-matrix (1.19, other) -     🔗  ✅ 4748  ❌ 0  ⏭ 0    🔗 See Details
integration-tests -     🔗  ✅ 55  ❌ 0  ⏭ 0    🔗 See Details
setup-environment -     🔗  N/A See Details
check-collector-module-version -     🔗  N/A See Details
lint-matrix (receiver-0) -     🔗  N/A See Details
lint-matrix (receiver-1) -     🔗  N/A See Details
lint-matrix (processor) -     🔗  N/A See Details
lint-matrix (exporter) -     🔗  N/A See Details
lint-matrix (extension) -     🔗  N/A See Details
lint-matrix (connector) -     🔗  N/A See Details
lint-matrix (internal) -     🔗  N/A See Details
lint-matrix (other) -     🔗  N/A See Details
check-codeowners -     🔗  N/A See Details
build-examples -     🔗  N/A See Details
checks -     🔗  N/A See Details
unittest (1.20) -     🔗  N/A See Details
unittest (1.19) -     🔗  N/A See Details
lint -     🔗  N/A See Details
cross-compile (darwin, amd64) -     🔗  N/A See Details
cross-compile (darwin, arm64) -     🔗  N/A See Details
cross-compile (linux, 386) -     🔗  N/A See Details
cross-compile (linux, amd64) -     🔗  N/A See Details
cross-compile (linux, arm) -     🔗  N/A See Details
cross-compile (linux, arm64) -     🔗  N/A See Details
cross-compile (linux, ppc64le) -     🔗  N/A See Details
cross-compile (windows, 386) -     🔗  N/A See Details
cross-compile (windows, amd64) -     🔗  N/A See Details
build-package (deb) -     🔗  N/A See Details
build-package (rpm) -     🔗  N/A See Details
windows-msi -     🔗  N/A See Details
publish-check -     🔗  N/A See Details
publish-dev -     🔗  N/A See Details
publish-stable -     🔗  N/A See Details

✅  prometheus-compliance-tests workflow has finished in 10 minutes 59 seconds (⚠️ 4 minutes 7 seconds more than main branch avg.) and finished at 22nd Mar, 2023.


Job Failed Steps Tests
prometheus-compliance-tests -     🔗  ✅ 21  ❌ 0  ⏭ 0    🔗 See Details

✅  load-tests workflow has finished in 16 minutes 16 seconds (⚠️ 4 minutes 17 seconds more than main branch avg.) and finished at 22nd Mar, 2023.


Job Failed Steps Tests
loadtest (TestTraceAttributesProcessor) -     🔗  ✅ 3  ❌ 0  ⏭ 0    🔗 See Details
loadtest (TestIdleMode) -     🔗  ✅ 1  ❌ 0  ⏭ 0    🔗 See Details
loadtest (TestMetric10kDPS|TestMetricsFromFile) -     🔗  ✅ 6  ❌ 0  ⏭ 0    🔗 See Details
loadtest (TestTraceNoBackend10kSPS|TestTrace1kSPSWithAttrs) -     🔗  ✅ 8  ❌ 0  ⏭ 0    🔗 See Details
loadtest (TestMetricResourceProcessor|TestTrace10kSPS) -     🔗  ✅ 12  ❌ 0  ⏭ 0    🔗 See Details
loadtest (TestTraceBallast1kSPSWithAttrs|TestTraceBallast1kSPSAddAttrs) -     🔗  ✅ 10  ❌ 0  ⏭ 0    🔗 See Details
loadtest (TestBallastMemory|TestLog10kDPS) -     🔗  ✅ 18  ❌ 0  ⏭ 0    🔗 See Details
setup-environment -     🔗  N/A See Details

✅  e2e-tests workflow has finished in 15 minutes 24 seconds and finished at 22nd Mar, 2023.


Job Failed Steps Tests
kubernetes-test (v1.26.0) -     🔗  N/A See Details
kubernetes-test (v1.25.3) -     🔗  N/A See Details
kubernetes-test (v1.24.7) -     🔗  N/A See Details
kubernetes-test (v1.23.13) -     🔗  N/A See Details

🔎 See details on Foresight

*You can configure Foresight comments in your organization settings page.

@mar4uk mar4uk force-pushed the lokiexporter-dots-to-underscores branch 3 times, most recently from 6d34372 to a578b3f Compare March 7, 2023 17:04
@mar4uk mar4uk marked this pull request as ready for review March 7, 2023 17:25
@mar4uk mar4uk requested a review from a team March 7, 2023 17:25
Copy link
Member

@gouthamve gouthamve left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@mar4uk mar4uk force-pushed the lokiexporter-dots-to-underscores branch from a578b3f to f659ef9 Compare March 8, 2023 11:23
# These lines will be padded with 2 spaces and then inserted directly into the document.
# Use pipe (|) for multiline entries.
subtext: |
Loki doesn't support label names containing dots. After normalization, all dots in label names are replaced with underscores
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shall we add a sentence telling users that they can drop relabeling from their configuration and that now the . -> _ will happen automatically.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thank you, fixed

That’s why to add Loki label based on `event.domain` OTLP attribute we need to specify two actions. The first one inserts a new attribute `event_domain` from the OTLP attribute `event.domain`. The second one is a hint for Loki, specifying that the `event_domain` attribute should be placed as a Loki label.
The same approach is applicable to placing Loki labels from resource attribute `service.name`.
Currently, Loki does not support label names with dots.
That's why lokiexporter normalizes label names to follow Prometheus label names standard before sending requests to Loki.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can point to pkg/translator/prometheus/README.md#Labels for more information on labels normalization.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added one more sentence with a link to pkg/translator/prometheus/README.md#Labels

@mar4uk mar4uk force-pushed the lokiexporter-dots-to-underscores branch from f659ef9 to 20adfa7 Compare March 10, 2023 14:07
@mar4uk mar4uk requested review from kovrus and removed request for jpkrohling and gramidt March 10, 2023 14:10
Copy link
Member

@kovrus kovrus left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

Copy link
Contributor

@evan-bradley evan-bradley left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me.

@mar4uk mar4uk requested a review from jpkrohling as a code owner March 15, 2023 10:47
@github-actions github-actions bot requested a review from gramidt March 15, 2023 10:48
@mar4uk mar4uk force-pushed the lokiexporter-dots-to-underscores branch from 6389a01 to 78c07f6 Compare March 15, 2023 10:52
Copy link
Contributor

@codeboten codeboten left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please resolve conflicts

@mar4uk mar4uk force-pushed the lokiexporter-dots-to-underscores branch from 78c07f6 to 93eb061 Compare March 17, 2023 09:16
@mar4uk
Copy link
Contributor Author

mar4uk commented Mar 20, 2023

@codeboten could you please take another look? conflicts are resolved

@mar4uk mar4uk force-pushed the lokiexporter-dots-to-underscores branch from 93eb061 to 363a087 Compare March 21, 2023 08:49
@gouthamve
Copy link
Member

Needs another rebase after #18532 unfortunately @mar4uk 😭

@mar4uk mar4uk force-pushed the lokiexporter-dots-to-underscores branch from 363a087 to bba3767 Compare March 21, 2023 13:37
@mar4uk mar4uk force-pushed the lokiexporter-dots-to-underscores branch from bba3767 to 5b1ba89 Compare March 22, 2023 09:43
@codeboten codeboten merged commit 0c68819 into open-telemetry:main Mar 22, 2023
@mar4uk mar4uk deleted the lokiexporter-dots-to-underscores branch March 22, 2023 15:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants