apm: Document sampling.tail.discard_on_write_failure config #1453
+23
−2
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
isaacaflores2
commented
May 22, 2025
| @@ -53,6 +53,10 @@ If a setting is not supported by {{ech}}, you will get an error message when you | |||
| Some settings that could break your cluster if set incorrectly are blocklisted. The following settings are generally safe in cloud environments. For detailed information about APM settings, check the [APM documentation](/solutions/observability/apm/configure-apm-server.md). | |||
| :::: | |||
|
|
|||
| ### Version 9.1+ [ec_version_9_1] | |||
carsonip
approved these changes
May 22, 2025
| @@ -85,6 +85,18 @@ Policies map trace events to a sample rate. Each policy must specify a sample ra | |||
| | APM Server binary | `sampling.tail.policies` | | |||
| | Fleet-managed | `Policies` | | |||
|
|
|||
| ### Discard On Write Failure [sampling-tail-discard-on-write-failure-ref] | |||
|
|
|||
| Defines the indexing behavior when trace events fail to be written to storage (e.g. when the storage limit is reached). When set to `false`, traces will be indexed, significantly increasing the indexing load. When set to `true`, traces will be discarded, there will be data loss potentially resulting in broken traces. | |||
There was a problem hiding this comment.
Suggested change
| Defines the indexing behavior when trace events fail to be written to storage (e.g. when the storage limit is reached). When set to `false`, traces will be indexed, significantly increasing the indexing load. When set to `true`, traces will be discarded, there will be data loss potentially resulting in broken traces. | |
| Defines the indexing behavior when trace events fail to be written to storage (e.g. when the storage limit is reached). When set to `false`, traces will be indexed regardless of the configured sample rate in policies, significantly increasing the indexing load. When set to `true`, traces will be discarded, there will be data loss potentially resulting in broken traces. |
nit on description. Trying to make the implication clear. Feel free to change.
There was a problem hiding this comment.
Thanks. I added a note to specify we bypass sampling
isaacaflores2
commented
May 22, 2025
| @@ -146,7 +146,7 @@ Due to [OpenTelemetry tail-based sampling limitations](/solutions/observability/ | |||
|
|
|||
| Tail-based sampling (TBS), by definition, requires storing events locally temporarily, such that they can be retrieved and forwarded when a sampling decision is made. | |||
|
|
|||
| In an APM Server implementation, the events are stored temporarily on disk instead of in memory for better scalability. Therefore, it requires local disk storage proportional to the APM event ingestion rate and additional memory to facilitate disk reads and writes. If the [storage limit](/solutions/observability/apm/tail-based-sampling.md#sampling-tail-storage_limit-ref) is insufficient, sampling will be bypassed. | |||
| In an APM Server implementation, the events are stored temporarily on disk instead of in memory for better scalability. Therefore, it requires local disk storage proportional to the APM event ingestion rate and additional memory to facilitate disk reads and writes. If the [storage limit](/solutions/observability/apm/tail-based-sampling.md#sampling-tail-storage_limit-ref) is insufficient, trace events will be indexed or discarded based on the [discard on write failure](/solutions/observability/apm/tail-based-sampling.md#sampling-tail-discard-on-write-failure-ref) configuration. | |||
There was a problem hiding this comment.
I found one other place where the storage limit and sampling bypass was mentioned. Updated to describe the new behavior
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Document
sampling.tail.discard_on_write_failureconfig.I sourced the config explanation from here please let me know if the description is incorrect or unclear in any way.
Updated pages can be found in the docs preview here:
Checklist
Related issues
Part of elastic/apm-server#15330