[ML] Improve audit message for hard_limit

[pheyos]

When ML jobs are hitting the hard_limit an audit message is generated. This message contains details about where the limit was hit. But currently, this number is not very helpful, because it only reflects the latest model_bytes of the job, which does not include the memory of the next stage of the analysis.

Example:

• Model memory limit setting: 6mb
• Reported model_bytes: 3.8mb
• Job audit message: Job memory status changed to hard_limit at 1.4mb [...]

This could be confusing for the user.

Feature request: Adjust the audit message to better explain why the hard_limit is hit.

[elasticmachine]

Pinging @elastic/ml-core

[droberts195]

@grabowskit saw the same thing with the Iowa liquor store data on 7th September last year.

I imagine it's a common problem when a large number of new split field values are seen for the first time in the same bucket.

In order to make the message sensible we need our model_size_stats to include not just current model_bytes but also something like max_predicted_model_bytes. It would populate that new field with the highest estimate it had ever made of how much memory it would need to do the next analysis step, even if as a result of making that estimate it decided not to do that analysis step and instead put the job into the hard_limit state.

@edsavage at some point (it's not mega urgent) please could you do a more in depth assessment of how we could feed back this extra statistic from the autodetect process into the model_size_stats?