[Breaking change] Switch logging to ECS-JSON layout by default

[Mpdreamz]

Change description

Which release will ship the breaking change?

TBD, The sooner the better but we'd have to have a deprecation period.

We could potentially evaluate enabling it by default on Cloud as an opaque change to the user in Isolation.

Is this a Kibana or Elasticsearch breaking change?

Kibana

Describe the change. How will it manifest to users?

Switching the default logging format to ECS is a move several of Elastic products are making with 8.0.

Switching to ECS structured logs helps unifying our log collection and aids with log correlation. E.g logs with trace.id can be used to enable quick navigation from Discover to Logs UI to APM Trace overview in any direction.

How many users will be affected?

Anyone manually ingesting Kibana logs.

Users using the kibana module from Filebeat to ingest kibana logs.

Are there any edge cases?

Unknown if any of the open items #60391 influences a decision.

Cross links

elastic/elasticsearch#47105
elastic/apm-server#3829
elastic/beats#15544

[elasticmachine]

Pinging @elastic/kibana-core (Team:Core)

[mshustov]

Reading #52226, I believe it should have been done there. But we stopped at just refactoring the JSON format to be ECS-compliant.
I'm not sure we can switch the default right now without a deprecation period (at least via Upgrade Assistant UI).

However, I talked to @pgomulka about how Elasticsearch approached the breaking change.
The Elasticsearch team is going to introduce the breaking change without a deprecation message, the change will be reflected in the migration guide only (elastic/elasticsearch#79146)

Right now, only Kibana and Beats don't use ECS-JSON by default https://github.com/elastic/dev/issues/1323
Beats team is working on the matter in elastic/beats#15544 @andresrc is that planned for v8.0?

@lukeelmers @kobelb What do you think about introducing this breaking change at such a late stage respecting Make it minor initiative?
If we aren't comfortable with adding a breaking change, we can document how to make the ECS-JSON format the default and plan the work for the next major release.

[lukeelmers]

I'm not sure we can switch the default right now without a deprecation period (at least via Upgrade Assistant UI).

What do you think about introducing this breaking change at such a late stage

If making this the default behavior is the direction the rest of the stack is going (including Elasticsearch), then I think it makes sense for us to do so as well.

I'm not super concerned about introducing the breaking change without a deprecation period if we are able to squeeze it into 8.0 (we should discuss feasibility of this), because:

1. We are already breaking all of the logging formats in 8.0 anyway (both pattern & json layouts), so this will mostly require updating documentation to make the changes clear to users
2. As the json layout is already ECS-compliant, it shouldn't be an enormous effort for us to update the default

[kobelb]

I wish we had a time machine to go back in time and know that when we switch to the new logging format we should've used ECS-JSON intsead of just JSON. Since we don't, I think that you all are proposing is the best course of action given the tough position we're in.

I've asked @lukeelmers to send an email to @mfinkle summarizing the breaking change we're proposing making to get his official approval.

[lukeelmers]

We've confirmed we're good to go on this (thanks @mfinkle). I'll take a look at it. I think we'll need to split this into a couple of pieces:

1. PR to 7.x to check for users who are (a) using the new logging configs, and (b) using the default layout, so that we can log a deprecation warning and surface instructions in the Upgrade Assistant.
2. PR to 8.0/master to switch the default layout to use json instead of pattern.
   • As a part of this I will see if it's possible to keep pattern layout as the default in dev mode, as most devs are working with the default settings & json is much harder to read in the console. If necessary this could also be a separate PR.
3. PR to add clearer documentation (we discussed the idea of a "logging upgrade guide") to help provide a high-level overview for users. Currently we have lots of documentation on the new logging system, but it is fairly detailed & low-level.

when we switch to the new logging format we should've used ECS-JSON intsead of just JSON

@kobelb To clarify on this, the new JSON format is ECS compliant. The issue is just that it isn't our default layout. (This was done to be consistent with the legacy configuration, where you had to explicitly say logging.json: true). So in this case I think we're mostly concerned with changing that default.