[aws][guardduty] System test failing in 8.19

[brijesh-elastic]

After updating the minimum Kibana version to ^8.19.0 or ^9.1.0 for AWS, the system tests for GuardDuty are failing.
Error

test case failed: one or more errors found while examining elastic-agent.logs1426453570: [0] found error "Unit state changed httpjson-default-httpjson-guardduty-305bec5b-8a77-4cbd-9c43-28473328f153 (HEALTHY->DEGRADED): failed to execute template last_execution_datetime: template: :1:28: executing \"\" at <.last_response.body.findings>: map has no entry for key \"findings\""

Upon investigating further, it appears that the error arises while setting cursor.last_execution_datetime.

• The expected behavior from the GuardDuty data collection was that the cursor should only be set after we've published the events, which occurs after the chain request call.
• However, it is currently being set before the chain request call and after the first/parent call. At that point, findings will not be present in the .last_response.body.

According to the httpjson documentation, the cursor should only be set after the events are published. (See the first paragraph in the cursor section). Also, events will only be published after the last chain call. (See the request_life_cycle section)

[elasticmachine]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

[efd6]

This is a consequence of the addition of fleet health status updates in the httpjson input. This happens here. The logic that leads to this though can be made more precise.

It's worth noting that the documentation stating that the cursor is not set until the events are is true since the cursor state in the store is not set until the corresponding events are published — though it is understandable that cursor state evaluation could be the reader's interpretation when they see the word "set". The documentation is silent as to when/whether cursor template evaluations happen.

[efd6]

This was fixed by #14525