[Stack 8.16.0-SNAPSHOT] [ti_crowdstrike] Failing test daily: system test: (elastic-agent logs - default) in ti_crowdstrike.intel

[elastic-vault-github-plugin-prod]

• Stack version: 8.16.0-SNAPSHOT
• Package: ti_crowdstrike
• Failing test: system test: (elastic-agent logs - default)
• DataStream: intel
• Owners:
  • @elastic/security-service-integrations

Failure:

test case failed: one or more errors found while examining elastic-agent.logs3969851264: [0] found error "Unit state changed cel-default-cel-ti_crowdstrike-3c1ad043-6455-4b07-baf1-383d741a7756 (HEALTHY->DEGRADED): single event error object returned by evaluation: {\"error\":{\"code\":\"404\",\"id\":\"404 Not Found\",\"message\":\"GET:404 Not Found (404)\"}}"

First build failed: https://buildkite.com/elastic/integrations/builds/14060

Latest failed builds:

• https://buildkite.com/elastic/integrations/builds/14065
• https://buildkite.com/elastic/integrations/builds/14073
• https://buildkite.com/elastic/integrations/builds/14158
• https://buildkite.com/elastic/integrations/builds/14203

[efd6]

Looking at the set of failures in these, I see (mod the tenable_io issue, which is addressed by #10940) over a wide range of packages: no hits; changes in processor behaviour; missing field definitions — present in the definitions files; and failed CEL evaluations — in the case where the error is reported a 404. This looks very much to me like a global CI failure rather than a specific cyberarkpas failure.

All of the elastic-agent logs that are retained in the builders' stores have staled out, so to investigate this further we will need to see a new failure.

Same symptoms as #10620

[andrewkroh]

Let's close it then. This hasn't reoccurred since July 31.