[Subscription basic] [rapid7_insightvm] Failing test daily: system test: (elastic-agent logs - default) in rapid7_insightvm.asset

[elastic-vault-github-plugin-prod]

• Stack version: Same as in Pull Request builds
• Subscription: basic
• Package: rapid7_insightvm
• Failing test: system test: (elastic-agent logs - default)
• DataStream: asset
• Owners:
  • @elastic/security-service-integrations

Failure:

test case failed: one or more errors found while examining elastic-agent.logs4045198688: [0] found error "Unit state changed httpjson-default-httpjson-rapid7_insightvm-37956309-1771-4a45-8025-80716d6cdbf4 (HEALTHY->DEGRADED): failed to execute template cursor: the template result is empty"

First build failed: https://buildkite.com/elastic/integrations/builds/27579

Latest 5 failed builds:

• https://buildkite.com/elastic/integrations/builds/30712
• https://buildkite.com/elastic/integrations/builds/30900
• https://buildkite.com/elastic/integrations/builds/31217
• https://buildkite.com/elastic/integrations/builds/31684
• https://buildkite.com/elastic/integrations/builds/31751

[navnit-elastic]

Related discussion in #14491 (comment).

[navnit-elastic]

This should be closed by elastic/beats#45361. We should wait for some time before we close this to monitor few new builds that should be passing.

[navnit-elastic]

This should be closed by elastic/beats#45361. We should wait for some time before we close this to monitor few new builds that should be passing.

After having look at elastic/beats#45361, It seems the change is for cursor and not updating Fleet health status for empty HTTPJSON cursor template value evaluations. In this case the empty value occurs in pagination. The solution could be adding do_not_log_failure to not log the failure.

do_not_log_failure if set to true a fail_on_template_error will not be logged except at DEBUG level. This should be used when template failure is expected in normal operation as control flow.

[navnit-elastic]

blacked on elastic/beats#45664.

[navnit-elastic]

We need a way to stop pagination without updating the fleet health status to degraded, which will be addressed after PR elastic/beats#46332 is merged.