[Breaking change] Audit logging events have changed

[thomheymann]

Change description

Which release will ship the breaking change?

8.0

Describe the change. How will it manifest to users?

Dropped support for legacy audit event. These have been replaced with audit events in ECS format which simplify ingestion into the stack, analysis with SIEM solution and provide more context and meta data.

The following events will not longer be logged: https://www.elastic.co/guide/en/kibana/current/xpack-security-audit-logging.html#_audit_event_types

How many users will be affected?

Anyone who is using the previous Kibana audit logger.

What can users do to address the change manually?

The ECS audit logger is controlled by the same xpack.security.audit.enabled setting as the legacy audit logger so no configuration needs to be changed and new event will automatically get logged out in the new format, if audit logging was previously enabled.

However, if users have alerts setup in Kibana or external systems those will need to be updated to look for the new ECS audit events instead:

• saved_objects_authorization_success - Instead filter events by event.category=database and event.outcome=success|unknown
• saved_objects_authorization_failure - Instead filter events by event.category=database and event.outcome=failure - The failure reason will be captured in the error property

A full list of events can be found here: https://www.elastic.co/guide/en/kibana/master/xpack-security-audit-logging.html#xpack-security-ecs-audit-logging

How could we make migration easier with the Upgrade Assistant?

Upgrade Assistant could check if xpack.security.audit.enabled is set, and if so add a "warning" message with the migration guidance shared above. Users could dismiss the warning if it didn't apply to them or if they've addressed it.

Are there any edge cases?

n/a

Test Data

n/a

Cross links

n/a

[elasticmachine]

Pinging @elastic/es-ui (Team:Elasticsearch UI)

[cjcenizal]

@thomheymann What if Upgrade Assistant checked if xpack.security.audit.enabled was set, and if so then it added an item to a "Warnings" section with the migration guidance you shared? Users could dismiss the warning if they decided it didn't apply to them or if they've addressed it. Would affected users find that helpful?

[thomheymann]

@cjcenizal I didn't know that we can surface messaging within the migration assistant. It's a great idea - I think that would be really useful for affected users. Especially if we list out what events they can consume instead.

[cjcenizal]

@thomheymann Sounds good! Thanks for the validation. We have the opportunity to redesign the Upgrade Assistant for an awesome 8.0 upgrade experience, so the sky's the limit. Anything you can think of that would help the user is fair game when considering new features we should add to UA.

[alisonelizabeth]

FYI I updated the issue title and How could we make migration easier with the Upgrade Assistant? section with CJ's suggestion. @thomheymann feel free to add anything else you think might be helpful.

[alisonelizabeth]

Hi @thomheymann! I'm going to remove the Elasticsearch UI team label and replace with security. The Upgrade Assistant will no longer be responsible for adding individual deprecation logic. Please consider using the deprecations service (#94845) provided by core to register this deprecation instead. All registered deprecations will be displayed in the Upgrade Assistant (to be implemented via #97159). Feel free to reach out to myself or the core team with any questions!