Performance regression related to DatafeedTimingStats doc persistence.

[przemekwitek]

Most likely the DatafeedTimingStats doc is persisted too often. This issue aims to verify and fix that.

From @dolaru:

we noticed a significant increase in some of our anomaly detection & data frame analytics job runs in CI.
the changes have been spotted starting with mini-stack build number 5302 in master. Looking at the changes that this build introduced, it seems that this PR might be the culprit: e2bdd43
The increase has been spotted in 7.x (7.4.0) and master (8.0.0) on both Linux and Windows machines.

[elasticmachine]

Pinging @elastic/ml-core

[wwang500]

Hi, @przemekwitek , after the fix #44803, we see a big improvement for overall job duration, except that there are still significant increase for 20 + anomaly detection jobs. like:

• "r02c", job duration is increased from 9s(build 8.0.0-5601) to 171s(build 8.0.0-5392), increased 1655 %,
• "median", job duration is increased from 23s(build 8.0.0-5601) to 192s (build 8.0.0-5392), increased 723%,
• "r02a", job duration is increased from 15s(build 8.0.0-5601) to 122s (build 8.0.0-5392), increased 705%,
• "r02b", job duration is increased from 17s(build 8.0.0-5601) to 74s (build 8.0.0-5392), increased 329%,
• "dns_tunneling1", job duration is increased from 13s(build 8.0.0-5601) to 51s (build 8.0.0-5392), increased 287%,

dataset_name	job_duration (ms) - build 5601	job_duration (ms) - latest build 5392	Increased %
r02c	9769	171476	1655.307606
median	23407	192795	723.6638612
r02a	15191	122405	705.7731552
r02b	17291	74241	329.3620959
dns_tunneling1	13177	51060	287.4933596
low_distinct_count_by_over	20777	65626	215.8588824

Could you please take a look at those as well? thanks in advanced!

[przemekwitek]

Could you please take a look at those as well? thanks in advanced!

I'm preparing another fix that is supposed to solve this performance problem.

[wwang500]

@przemekwitek, would you mind provide an ETA for the coming fix? QA needs to decide if we should temporarily mute the job duration regression failures or not. Thanks.

[przemekwitek]

The fix is currently under review: #44940
Hopefully I'll be able to merge it in today or tomorrow.

[przemekwitek]

The fix to master (#44940) as well as backport to 7.x (#45079) are both merged in.

Closing the issue. Please let me know in case the problem reoccurs.