Log HTTP requests, responses

[azasypkin]

Currently Kibana is able to log requests, corresponding responses and various operations data (CPU, memory, disk, and other metrics). It's done with the help of even-better HapiJS plugin (fork of good plugin). As far as I can tell this information is only logged with logging.json: true.

Also there is additional logging.filter configuration value that allows Kibana to remove/censor some sensitive data from logs (e.g. authorization or cookie headers).

In the new-platform we're not going to use HapiJS so we'll have to handle this differently. So let's start from the following questions first:

• Are we still interested in all this data or is there anything we don't need anymore?
• What additional "service" data we'd like to log, but don't do right now? It's a good time for enhancement requests.
• Is "censoring" used primarily for authorization and cookie headers (i.e. only request/response use case)?

Any thoughts?

/cc @elastic/kibana-platform @elastic/kibana-operations

---

Tasks:

• Add a logging.filter config Implement filtering mechanism in the Logging service #57547
• Log ops metrics [logging/metrics] Provide ability to log ops data from core metrics service #86162
• Add request/response data

[epixa]

Are we still interested in all this data or is there anything we don't need anymore?

To ensure that this can wrong alongside the existing core, I think we need to preserve all of this data in exactly the same format as before.

What additional "service" data we'd like to log, but don't do right now? It's a good time for enhancement requests.

I'm not against new data or anything, but if we add new data in the new platform, then we'll need to do the same in the old core until that aspect is moved entirely over to the new platform, so I recommend avoiding it.

Is "censoring" used primarily for authorization and cookie headers (i.e. only request/response use case)?

Those are the only things I can think of

[azasypkin]

To ensure that this can wrong alongside the existing core, I think we need to preserve all of this data in exactly the same format as before.

Yep, filed #13412 to track that.

I'm not against new data or anything, but if we add new data in the new platform, then we'll need to do the same in the old core until that aspect is moved entirely over to the new platform, so I recommend avoiding it.

Yeah, the plan is to carefully and selectively replace current plugins/sub-systems/features with counterparts from the new platform one by one (e.g. logging). To avoid double work we'll be introducing breaking changes/new behavior only for already incorporated stuff. Here I just wanted to collect ideas in advance :)

Those are the only things I can think of

Good, thanks.

[epixa]

@azasypkin Anything else we need from this issue? We're moving forward with hapi, so that makes part of this moot. The rest seems relevant, but it also seems like we have a decision.

[azasypkin]

Hmm, I think we need to do this eventually anyway. The fact that we're going to use Hapi in new platform doesn't change our intention to rely on our own logging system and get rid of good/even-better, right?

At some point we'll reach feature parity between new and old logging systems and just get rid of Hapi-based logging (aka even-better) and switch to new logging. At least that was the idea initially.

[kobelb]

Is "censoring" used primarily for authorization and cookie headers (i.e. only request/response use case)?

As far as I'm aware from the even-better docs, these filters only apply to the response events from HapiJS

filter can be used to remove potentially sensitive information (credit card numbers, social security numbers, etc.) from the log payloads before they are sent out to reporters. This setting only impacts response events and only if payloads are included via requestPayload and responsePayload

I'm dealing with a situation where a user has a custom auth header showing up in their logs, and they need to filter it out and I recommended taking advantage of logging.filter to do so. I'd be helpful to see a similar facility in place if we switch the logging infrastructure in the new platform.

[azasypkin]

I'd be helpful to see a similar facility in place if we switch the logging infrastructure in the new platform.

Yep, indeed, thanks for the input!

[tsullivan]

Hi, is this advice still relevant to users that want to have the username from Auth header logged? https://discuss.elastic.co/t/how-can-i-setup-kibana-server-logs-for-capturing-the-user-logged-in-and-dashboards-viewed-info/174575/2?u=tsullivan

The way I've dealt with this requirement was to have a reverse proxy in front of Kibana that logs the request, including the URI and the auth username. You can have the proxy's logs ingested into Elasticsearch using Filebeat.

[azasypkin]

Hi, is this advice still relevant to users that want to have the username from Auth header logged? https://discuss.elastic.co/t/how-can-i-setup-kibana-server-logs-for-capturing-the-user-logged-in-and-dashboards-viewed-info/174575/2?u=tsullivan

Hey, sorry for the late reply! I guess you can also achieve that with something like this:

logging:
  filter.authorization: none # it's `remove` by default
  json: true