[ML] AIOps: Tweak log rate changes in log rate analysis results table. #188648

walterra · 2024-07-18T13:03:41Z

Summary

Part of #187684.

This moves functions related to log rate changes to the @kbn/aiops_log_rate_analysis package.

getLogRateAnalysisType was renamed to getLogRateAnalysisTypeForHistogram to indicate its use with histogram data.
getLogRateAnalysisTypeForCounts was added for cases where we don't have the histogram data available but just the doc counts for baseline an deviation time ranges. This isn't used yet as of this PR but will be in a follow up in combination with the o11y AI assistant.
getSwappedWindowParameters is a helper to consolidate inline code that's used to swap baseline and deviation when we detected a dip in log rate.
Rounding for the log rate change messages was tweaked. Changes below 10x will now be rounded to one digit to avoid messages like 1x increase.
Tweaked/Shortened the message for 0 in baseline or deviation to just 45 up from 0 in baseline / down to 0 from 45 in baseline.

Checklist

Unit or functional tests were updated or added to match the most common scenarios
This was checked for breaking API changes and was labeled appropriately

elasticmachine · 2024-07-18T13:53:24Z

Pinging @elastic/ml-ui (:ml)

peteharverson

Other than a suggestion for a slight edit to the text, tested and LGTM

x-pack/packages/ml/aiops_log_rate_analysis/get_log_rate_change.ts

…fix'

walterra · 2024-07-19T15:27:52Z

I noticed a bug how the log rate change text was calculated and updated. Because we always calculated it based on the window parameters selected in the chart, if you moved the brushes after the analysis was done, it would also update the text in the log rate change columns. I fixed it by moving currentAnalysisWindowParameters to the redux state to make it available for the columns so the calculation is always done on the values corresponding on what has been analysed. For clarity, I renamed the other windowParameters to chartWindowParameters to make the difference more obvious.

Before:

After:

Fixed in fb8a96d.

elasticmachine · 2024-07-19T16:16:31Z

💔 Build Failed

Buildkite Build
Commit: 244c96b

Failed CI Steps

Check Types

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
`aiops`	634	636	+2
`dataVisualizer`	791	794	+3
total			+5

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`aiops`	554.7KB	555.5KB	+774.0B

Unknown metric groups

API count

id	before	after	diff
`@kbn/aiops-log-rate-analysis`	54	66	+12

History

💔 Build #222629 failed 62585b4
💔 Build #222604 failed 1ff0b8c

cc @walterra

alvarezmelissa87 · 2024-07-22T19:09:44Z

x-pack/packages/ml/aiops_log_rate_analysis/api/stream_reducer.ts

@@ -80,7 +85,12 @@ export const logRateAnalysisResultsSlice = createSlice({
    resetGroups: (state) => {
      state.significantItemsGroups = [];
    },
-    resetAll: () => getDefaultState(),
+    // reset the results but keeps


Nit: unfinished comment explanation?

Thanks for spotting this! Fixed in 6941aa7.

alvarezmelissa87

LGTM ⚡

kibana-ci · 2024-07-23T08:12:36Z

💛 Build succeeded, but was flaky

Buildkite Build
Commit: 6941aa7

Failed CI Steps

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
`aiops`	634	636	+2
`dataVisualizer`	791	794	+3
total			+5

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`aiops`	554.9KB	555.6KB	+774.0B

Unknown metric groups

API count

id	before	after	diff
`@kbn/aiops-log-rate-analysis`	54	66	+12

History

💛 Build #222821 was flaky 4feb8d6

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @walterra

…from analysis (not just grouping) (#188913) ## Summary Part of #187684. So far the popover to filter fields was only available when grouping was enabled. This PR updates the behavior so it's available all the time and can be used to exclude field candidates from the analysis. If we detect the index to be based on an ECS schema, we auto-select a set of predefined fields. Changes in this PR: - Creates a new route `/internal/aiops/log_rate_analysis/field_candidates` to be able to fetch field candidates independent of the main streaming API call. - Fixes the code to consider "remaining" field candidates to also consider text field candidates. This was originally developed to allow to continue an analysis that errored for some reason. We use that option to also pass on the custom field list from the field selection popover. - Fetching the field candidates is done in a new redux slice `logRateAnalysisFieldCandidatesSlice` using an async thunk. - Filters the list of field candidates by a predefined field of allowed fields when an ECS schema gets detected. - Renames `fieldCandidates` to `keywordFieldCandidates` for clearer distinction against `textFieldCandidates`. - Refactors `getLogRateAnalysisTypeForCounts` args to a config object. - Bump the API version for the full log rate analysis to version 3. We missed bumping the version in #188648. This update manages proper versioning between v2 and v3, also the API integration tests cover both versions. [aiops-log-rate-analysis-fields-filter-0001.webm](https://github.com/user-attachments/assets/e3ed8d5b-f01c-42ef-8033-caa7135b8cc0) ### Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [ ] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed - [x] This was checked for breaking API changes and was [labeled appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)

reuse getLogRate()

3d6ed72

walterra self-assigned this Jul 18, 2024

walterra added 2 commits July 18, 2024 15:12

add jsdoc

ef0e9c5

jest unit tests

37110c4

walterra added release_note:skip Skip the PR/issue when compiling release notes :ml Feature:ML/AIOps ML AIOps features: Change Point Detection, Log Pattern Analysis, Log Rate Analysis v8.16.0 labels Jul 18, 2024

walterra requested review from alvarezmelissa87 and peteharverson July 18, 2024 13:53

walterra marked this pull request as ready for review July 18, 2024 13:53

walterra requested a review from a team as a code owner July 18, 2024 13:53

peteharverson approved these changes Jul 18, 2024

View reviewed changes

x-pack/packages/ml/aiops_log_rate_analysis/get_log_rate_change.ts Outdated Show resolved Hide resolved

walterra and others added 5 commits July 19, 2024 14:17

Merge branch 'main' into ml-aiops-log-rate-change-fixes

261bc98

fix wording

efba653

[CI] Auto-commit changed files from 'node scripts/lint_ts_projects --…

1ff0b8c

…fix'

x-pack/.i18nrc.json

62585b4

fix chart window parameters propagating to log rate change columns

fb8a96d

fix assertion

244c96b

walterra added 2 commits July 22, 2024 10:43

fix export

8378f03

Merge branch 'main' into ml-aiops-log-rate-change-fixes

4feb8d6

alvarezmelissa87 reviewed Jul 22, 2024

View reviewed changes

alvarezmelissa87 approved these changes Jul 22, 2024

View reviewed changes

walterra added 2 commits July 23, 2024 08:21

Merge branch 'main' into ml-aiops-log-rate-change-fixes

910f964

fix comment for resetResults

6941aa7

walterra added the bug Fixes for quality problems that affect the customer experience label Jul 23, 2024

walterra mentioned this pull request Jul 23, 2024

[ML] AIOps: 8.16 Log Rate Analysis improvements. #187684

Closed

walterra merged commit dffc044 into elastic:main Jul 23, 2024
23 checks passed

kibanamachine added the backport:skip This commit does not require backporting label Jul 23, 2024

walterra deleted the ml-aiops-log-rate-change-fixes branch July 23, 2024 08:25

walterra mentioned this pull request Jul 24, 2024

[ML] AIOps: Update fields filter popover to be able to filter fields from analysis (not just grouping) #188913

Merged

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ML] AIOps: Tweak log rate changes in log rate analysis results table. #188648

[ML] AIOps: Tweak log rate changes in log rate analysis results table. #188648

walterra commented Jul 18, 2024 •

edited

Loading

elasticmachine commented Jul 18, 2024

peteharverson left a comment

walterra commented Jul 19, 2024 •

edited

Loading

elasticmachine commented Jul 19, 2024 •

edited

Loading

API count

alvarezmelissa87 Jul 22, 2024

walterra Jul 23, 2024

alvarezmelissa87 left a comment

kibana-ci commented Jul 23, 2024

API count

[ML] AIOps: Tweak log rate changes in log rate analysis results table. #188648

[ML] AIOps: Tweak log rate changes in log rate analysis results table. #188648

Conversation

walterra commented Jul 18, 2024 • edited Loading

Summary

Checklist

elasticmachine commented Jul 18, 2024

peteharverson left a comment

Choose a reason for hiding this comment

walterra commented Jul 19, 2024 • edited Loading

elasticmachine commented Jul 19, 2024 • edited Loading

💔 Build Failed

Failed CI Steps

Metrics [docs]

Module Count

Async chunks

API count

History

alvarezmelissa87 Jul 22, 2024

Choose a reason for hiding this comment

walterra Jul 23, 2024

Choose a reason for hiding this comment

alvarezmelissa87 left a comment

Choose a reason for hiding this comment

kibana-ci commented Jul 23, 2024

💛 Build succeeded, but was flaky

Failed CI Steps

Metrics [docs]

Module Count

Async chunks

API count

History

walterra commented Jul 18, 2024 •

edited

Loading

walterra commented Jul 19, 2024 •

edited

Loading

elasticmachine commented Jul 19, 2024 •

edited

Loading