[Entity Analytics][9.0] Remove all legacy risk engine code and features #201810

hop-dev · 2024-11-26T15:19:02Z

Summary

Closes https://github.com/elastic/security-team/issues/11253
Breaking change proposal: https://github.com/elastic/dev/issues/2822

The host and user risk scoring modules or "legacy risk engine" as we often call it internally, has been superseded since v8.10.0 by the risk engine. We submitted a breaking change proposal for v9.0.0 to remove all support to this legacy approach which was approved.

In 8.18 users will be given a warning if they are still using the legacy risk engine and directed to upgrade, this is implemented in #202775.

Changes

we previously used the presence of the risk index to decide whether to show the "enable risk engine" button instead of the risk score table, now that we only have only one risk scoring approach, I have changed this to use the risk engine status API, if the risk engine has ever been installed we show the table.

Deletions

all code related to displaying legacy risk score
we no longer enrich alerts with legacy risk score
all code related to upgrading to the "new" risk engine
telemetry on the ml_risk_score* indices
all internal APIs related to the legacy risk engine

… when engine has been installed

hop-dev · 2024-12-02T15:46:01Z

/ci

hop-dev · 2024-12-02T16:17:58Z

/ci

hop-dev · 2024-12-03T09:56:46Z

/ci

hop-dev · 2024-12-03T10:48:38Z

/ci

… into delete-legacy-risk-engine

angorayc

Thanks for removing the unused code!

...ty_solution_cypress/cypress/e2e/entity_analytics/dashboards/enable_risk_score_redirect.cy.ts

MadameSheema · 2025-01-09T10:20:06Z

...y_solution_cypress/cypress/e2e/entity_analytics/dashboards/entity_analytics/risk_score.cy.ts

-        cy.get(ENABLE_HOST_RISK_SCORE_BUTTON).should('be.visible');
-
-        cy.get(ENABLE_USER_RISK_SCORE_BUTTON).should('be.visible');
+      it('shows enable risk button', () => {


It would be great if we can invest time to try to unskip this spec file :)

MadameSheema · 2025-01-09T10:38:04Z

x-pack/test/security_solution_cypress/cypress/e2e/entity_analytics/enrichments.cy.ts

-  });
-
-  after(() => {
-    cy.task('esArchiverUnload', { archiveName: 'risk_users' });
  });

  describe('Custom query rule', () => {


Same as before, it would be great to try to unskip the test :)

x-pack/test/security_solution_cypress/cypress/e2e/entity_analytics/host_details/risk_tab.cy.ts

x-pack/test/security_solution_cypress/cypress/e2e/entity_analytics/hosts/host_risk_tab.cy.ts

...ck/test/security_solution_cypress/cypress/e2e/entity_analytics/hosts/hosts_risk_column.cy.ts

nkhristinin

DE changes LGTM!

MadameSheema

Thanks for reviewing all the comments :)

This reverts commit 818cbdd.

hop-dev · 2025-01-10T17:23:42Z

@elasticmachine merge upstream

elasticmachine · 2025-01-10T19:05:18Z

💚 Build Succeeded

Buildkite Build
Commit: 49de90a

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
`securitySolution`	6541	6445	-96

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`securitySolution`	22.2MB	21.1MB	-1.1MB

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
`securitySolution`	88.2KB	87.3KB	-882.0B

Unknown metric groups

ESLint disabled line counts

id	before	after	diff
`securitySolution`	575	573	-2

References to deprecated APIs

id	before	after	diff
`securitySolution`	464	355	-109

Total ESLint disabled count

id	before	after	diff
`securitySolution`	659	657	-2

History

💔 Build #266000 failed 2adf1ae
💔 Build #265936 failed f27c789
💚 Build #264696 succeeded ad78b6f
💔 Build #264386 failed ed6cd1f
💔 Build #264300 failed 9743e35

cc @hop-dev

…es (elastic#201810)

…219858) ## Summary Closes #219490 Fix an error toast appearing when changing an entities asset criticality from the entity flyout. See video on issue above for more detail. We were seeing the risk score request be aborted and this was creating the error toast. - Do not toast if an abort error is thrown - Do not abort the request unless the component receives `skip:true` ### Test Steps 1. Use the security document generator `yarn start entity-store` command to load entities and enable the risk engine 2. Enable the entity store 3. from the entities table open the entity flyout 4. Assign/change asset criticality 5. observe the error is not present ### Why did the bug happen? The risk score component stops rendering because the conditions for aborting and searching differ. So we aborted the HTTP call but didn't search again. I believe this PR #201810 introduced the bug --------- Co-authored-by: machadoum <pablo.nevesmachado@elastic.co>

…lastic#219858) ## Summary Closes elastic#219490 Fix an error toast appearing when changing an entities asset criticality from the entity flyout. See video on issue above for more detail. We were seeing the risk score request be aborted and this was creating the error toast. - Do not toast if an abort error is thrown - Do not abort the request unless the component receives `skip:true` ### Test Steps 1. Use the security document generator `yarn start entity-store` command to load entities and enable the risk engine 2. Enable the entity store 3. from the entities table open the entity flyout 4. Assign/change asset criticality 5. observe the error is not present ### Why did the bug happen? The risk score component stops rendering because the conditions for aborting and searching differ. So we aborted the HTTP call but didn't search again. I believe this PR elastic#201810 introduced the bug --------- Co-authored-by: machadoum <pablo.nevesmachado@elastic.co> (cherry picked from commit 64d20bb)

…lastic#219858) ## Summary Closes elastic#219490 Fix an error toast appearing when changing an entities asset criticality from the entity flyout. See video on issue above for more detail. We were seeing the risk score request be aborted and this was creating the error toast. - Do not toast if an abort error is thrown - Do not abort the request unless the component receives `skip:true` ### Test Steps 1. Use the security document generator `yarn start entity-store` command to load entities and enable the risk engine 2. Enable the entity store 3. from the entities table open the entity flyout 4. Assign/change asset criticality 5. observe the error is not present ### Why did the bug happen? The risk score component stops rendering because the conditions for aborting and searching differ. So we aborted the HTTP call but didn't search again. I believe this PR elastic#201810 introduced the bug --------- Co-authored-by: machadoum <pablo.nevesmachado@elastic.co>

hop-dev added 9 commits November 26, 2024 14:30

no type errors :)

6367d11

fix index not found error handling

6534ff3

rename isEnabled to hasEngineBeenInstalled and only search risk score…

e32da15

… when engine has been installed

remove unused tests

98a5339

remove unused translations

bea1be4

tidy up unused test data

78f956a

found some more tests

8b9ab1d

translations

4612253

Merge branch 'main' into delete-legacy-risk-engine

8537861

hop-dev changed the title ~~Delete legacy risk engine~~ [Entity Analytics][9.0] Remove all legacy risk engine code Dec 2, 2024

hop-dev self-assigned this Dec 2, 2024

hop-dev added release_note:deprecation backport:skip This commit does not require backporting v9.0.0 Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. Team:Entity Analytics Security Entity Analytics Team labels Dec 2, 2024

hop-dev and others added 2 commits December 2, 2024 16:10

more type errors

715f219

[CI] Auto-commit changed files from 'yarn openapi:generate'

6ab5934

Remove legacy_risk_engine_disabled from InitRiskEngineResult and schema

4324559

remove legacy_risk_engine_status refs

d47b379

hop-dev added 6 commits December 5, 2024 13:15

delete legacy test data

d6a33a2

remove telemetry

22180bd

fix unit test

4b2eee5

fix user unit test

3be12a3

enrichment tests

2b7c514

remove unused types

549f7c6

Merge branch 'delete-legacy-risk-engine' of github.com:hop-dev/kibana…

ed6cd1f

… into delete-legacy-risk-engine

angorayc approved these changes Jan 6, 2025

View reviewed changes

Ikuni17 approved these changes Jan 7, 2025

View reviewed changes

vitaliidm removed their request for review January 7, 2025 09:33

hop-dev added 2 commits January 7, 2025 11:19

fix test snapshot

48f9902

Merge branch 'main' into delete-legacy-risk-engine

ad78b6f

hop-dev requested a review from MadameSheema January 8, 2025 20:13

MadameSheema reviewed Jan 9, 2025

View reviewed changes

Merge branch 'main' into delete-legacy-risk-engine

676c928

nkhristinin approved these changes Jan 10, 2025

View reviewed changes

hop-dev added 5 commits January 10, 2025 08:45

remove unused feature flag from test

818cbdd

remove extra describe

e81d87c

remove extra describe block

6ea6296

remove describe block

dd58dd9

Merge branch 'main' into delete-legacy-risk-engine

f27c789

MadameSheema approved these changes Jan 10, 2025

View reviewed changes

Revert "remove unused feature flag from test"

2adf1ae

This reverts commit 818cbdd.

Merge branch 'main' into delete-legacy-risk-engine

49de90a

hop-dev merged commit 80baa2c into elastic:main Jan 10, 2025
8 checks passed

hop-dev deleted the delete-legacy-risk-engine branch January 13, 2025 13:11

viduni94 pushed a commit to viduni94/kibana that referenced this pull request Jan 23, 2025

[Entity Analytics][9.0] Remove all legacy risk engine code and featur…

5d50e88

…es (elastic#201810)

ethack mentioned this pull request Apr 1, 2025

[Entity Analytics] Remove support for the legacy risk engine #188210

Closed

machadoum mentioned this pull request May 2, 2025

[Entity Analytics] [bug-fix] Do not abort risk score search request #219858

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[Entity Analytics][9.0] Remove all legacy risk engine code and features #201810

[Entity Analytics][9.0] Remove all legacy risk engine code and features #201810

Uh oh!

hop-dev commented Nov 26, 2024 •

edited

Loading

Uh oh!

hop-dev commented Dec 2, 2024

Uh oh!

hop-dev commented Dec 2, 2024

Uh oh!

hop-dev commented Dec 3, 2024

Uh oh!

hop-dev commented Dec 3, 2024

Uh oh!

angorayc left a comment

Uh oh!

Uh oh!

MadameSheema Jan 9, 2025

Uh oh!

MadameSheema Jan 9, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

nkhristinin left a comment

Uh oh!

MadameSheema left a comment

Uh oh!

hop-dev commented Jan 10, 2025

Uh oh!

elasticmachine commented Jan 10, 2025

ESLint disabled line counts

References to deprecated APIs

Total ESLint disabled count

Uh oh!

Uh oh!

Uh oh!

[Entity Analytics][9.0] Remove all legacy risk engine code and features #201810

[Entity Analytics][9.0] Remove all legacy risk engine code and features #201810

Uh oh!

Conversation

hop-dev commented Nov 26, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Uh oh!

hop-dev commented Dec 2, 2024

Uh oh!

hop-dev commented Dec 2, 2024

Uh oh!

hop-dev commented Dec 3, 2024

Uh oh!

hop-dev commented Dec 3, 2024

Uh oh!

angorayc left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

MadameSheema Jan 9, 2025

Choose a reason for hiding this comment

Uh oh!

MadameSheema Jan 9, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

nkhristinin left a comment

Choose a reason for hiding this comment

Uh oh!

MadameSheema left a comment

Choose a reason for hiding this comment

Uh oh!

hop-dev commented Jan 10, 2025

Uh oh!

elasticmachine commented Jan 10, 2025

💚 Build Succeeded

Metrics [docs]

Module Count

Async chunks

Page load bundle

ESLint disabled line counts

References to deprecated APIs

Total ESLint disabled count

History

Uh oh!

Uh oh!

Uh oh!

hop-dev commented Nov 26, 2024 •

edited

Loading