Closed
Description
openedon Dec 3, 2019
Overview
The current state of audit logging in Kibana is not sufficient for many users' needs. Kibana outputs only a few types of events, without much detail, in the same transport as regular log messages. This can be improved in many ways.
Enhancements in scope:
- More audit events and information regarding authentication -- e.g., log in and log out events
- More audit events for accessing objects
- Additional attributes for objects -- usernames, names, IPs, space IDs/names, object URLs, timestamps, authentication
- Additional information to differentiate specific user sessions
- Additional information to allow for correlation with Elasticsearch audit records
- Ability to include/exclude certain events and attributes
- Separate audit log transport with rotation capabilities
- Fail-safe to stop the Kibana process if audit records cannot be written
- Additional configuration to support all of the above enhancements
Current state vs. desired state...
Current state
Audit records in Kibana are displayed in plaintext like so:
log [23:26:50.059] [info][audit][saved_objects_authorization_success][security] jdoe authorized to get config
log [23:26:50.067] [info][audit][saved_objects_authorization_success][security] jdoe authorized to find index-pattern
If JSON output is enabled:
{
"type": "log",
"@timestamp": "2020-02-18T14:58:44-05:00",
"tags": [
"info",
"audit",
"security",
"saved_objects_authorization_success"
],
"pid": 38933,
"username": "jojo",
"action": "get",
"types": [
"config"
],
"args": {
"type": "config",
"id": "8.0.0",
"options": {}
},
"eventType": "saved_objects_authorization_success",
"message": "jojo authorized to get config"
}
{
"type": "log",
"@timestamp": "2020-02-18T14:58:44-05:00",
"tags": [
"info",
"audit",
"security",
"saved_objects_authorization_success"
],
"pid": 38933,
"username": "jojo",
"action": "find",
"types": [
"index-pattern"
],
"args": {
"options": {
"perPage": 1,
"page": 1,
"type": [
"index-pattern"
],
"search": "*",
"defaultSearchOperator": "OR",
"searchFields": [
"title"
],
"fields": [
"title"
]
}
},
"eventType": "saved_objects_authorization_success",
"message": "jojo authorized to find index-pattern"
}
Future state
Audit records should be written in a standard format (ECS), should contain more information about the event that occurred and who originated the action, and fields should be configurable to include more or less information. Such an audit record would look something like this:
{
"@timestamp": "2019-12-05T00:00:02.000Z",
"event": {
"action": "get config",
"category": "saved_objects_authorization",
"duration": 453,
"end": "2019-12-05T00:00:02.453Z",
"module": "security",
"outcome": "success",
"start": "2019-12-05T00:00:02.000Z"
},
"host": {
"id": "5b2de169-2785-441b-ae8c-186a1936b17d",
"ip": "34.56.78.90",
"hostname": "hostname"
},
"http": {
"request": {
"body": {
"bytes": 887,
"content": "Hello world"
},
"bytes": 1437,
"method": "get",
"referrer": "https://blog.example.com/"
}
},
"labels": {
"spaceId": "default"
},
"source": {
"address": "12.34.56.78",
"ip": "12.34.56.78"
},
"url": {
"domain": "www.elastic.co",
"full": "https://www.elastic.co:443/search?q=elasticsearch",
"path": "/search",
"port": "443",
"query": "q=elasticsearch",
"scheme": "https"
},
"user": {
"email": "john.doe@company.com",
"full_name": "John Doe",
"hash": "D30A5F57532A603697CCBB51558FA02CCADD74A0C499FCF9D45B...",
"sid": "2FBAF28F6427B1832F2924E4C22C66E85FE96AFBDC3541C659B67...",
"name": "jdoe",
"roles": [ "kibana_user" ]
},
"trace": {
"id": "8a4f500d"
}
}
Note: in the example above, the user.hash (a hash of the user.name field) would not be included by default; it would be an optional field that could be included if the user.name needed to be excluded for privacy reasons.
First Phase
Prerequisites (in progress):
- Format audit records in JSON using the Elastic Common Schema (ECS) [logging] Use Elastic Common Schema (ECS) #52226
- Modify Elasticsearch client to pass X-Opaque-Id header for unique events for correlation Pipe
X-Opaque-Idheader to AuditTrail logs and Elasticsearch API calls #62018 - Collect audit logs for ES client Add generic AuditTrail service #60119
- Implement server-side sessions Kibana Security to use Server Side Sessions #17870
Phase 1 implementation: #54836
Future Phase
- Enriching events with session ID
- Support for log rotation (prerequisite: [KP] Implement Log rotation appender #56291)
- Additional attributes such as IP address (Add client IP address to audit records #127481) and user profile ID (Add user profile ID to audit log events #125932)
- Fail-safe to stop Kibana process if audit records cannot be written Plugins can initiate Kibana graceful shutdown #60636
- Additional transport options (human-readable message formatting, multiple appenders)
- Support for including/excluding event attributes
- Include/exclude events based on attributes (such as saved object type)
- Additional configuration to support the above
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment