fix(kubernetes_logs): delaying applied new pod #13886
Conversation
✅ Deploy Preview for vector-project canceled.
|
Soak Test ResultsBaseline: 8f2ae47 ExplanationA soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core. The table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.3. The abbreviated table will be omitted if no interesting changes are observed. No interesting changes in throughput with confidence ≥ 90.00% and absolute Δ mean >= ±8.87%: Fine details of change detection per experiment.
|
sillent
changed the title
Soak Test ResultsBaseline: afcee9b ExplanationA soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core. The table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.3. The abbreviated table will be omitted if no interesting changes are observed. No interesting changes in throughput with confidence ≥ 90.00% and absolute Δ mean >= ±8.87%: Fine details of change detection per experiment.
|
|
Reading through the original issue, I don't believe this is a suitable fix. Anything that is based on the timing of an event is too fragile to be considered reliable in production workloads. Based on the comments that @jszwedko left, it seems like the correct solution should revolve around including a better unique identifier so that when we delete the metadata for the pod, we only delete the metadata for the previous pod, not the new pod that shares the same name. If you're willing to try to work up a solution based on that approach, feel free to do so in this PR. Otherwise, we would end up closing this one. |
|
No problem. I do not have a complete understanding of the architecture of this solution, if in your opinion the problem is not in this place, then the PR can be closed. |
if my guess described in problem #13467 is correct, then as a quick solution i suggest putting the event about the creation of a new pod in a delayed queue equal to the delayed queue for deleting a pod plus one second