[Bug] Package `v8.16.2` contains new rule versions without updates

[banderror]

Related to: elastic/kibana#201825 (review)

Summary

In Kibana, I upgraded the package with prebuilt rules from v8.16.2-beta.1 to v8.16.2 and got 64 detection rules that can be upgraded via the Security Solution UI. Out of those 64 rules, there were 26 rules that had only the version field bumped from something like 2 or 3 to something like 100+, and there were no other changes to rule fields in these newer versions of rules.

Here's an example of how such rules are displayed in the rule upgrade UI:

To Reproduce

On way to reproduce this when testing it with the local Kibana: elastic/kibana#201825 (review)

Expected Behavior

In the package with prebuilt rules, we shouldn't release a new rule version if it doesn't have any changes to any other rule parameters, compared to the previous version of the rule.

Problematic rules

Here's the full list of rules with updated version and no updates to any other fields:

• First Occurrence of Personal Access Token (PAT) Use For a GitHub User
• High Number of Cloned GitHub Repos From PAT
• First Occurrence of IP Address For GitHub Personal Access Token (PAT)
• GitHub App Deleted
• GitHub PAT Access Revoked
• Multiple Okta User Authentication Events with Client Address
• Multiple Okta User Authentication Events with Same Device Token Hash
• GitHub Owner Role Granted To User
• Okta User Sessions Started from Different Geolocations
• First Occurrence of IP Address For GitHub User
• GitHub User Blocked From Organization
• First Occurrence of User-Agent For a GitHub User
• Multiple Device Token Hashes for Single Okta Session
• First Occurrence GitHub Event for a Personal Access Token (PAT)
• First Occurrence of GitHub User Interaction with Private Repo
• First Occurrence of GitHub Repo Interaction From a New IP
• GitHub Protected Branch Settings Changed
• Member Removed From GitHub Organization
• First Occurrence of User Agent For a GitHub Personal Access Token (PAT)
• New GitHub App Installed
• First Occurrence of Private Repo Event from Specific GitHub Personal Access Token (PAT)
• High Number of Okta Device Token Cookies Generated for Authentication
• New GitHub Owner Added
• New User Added To GitHub Organization
• GitHub Repo Created
• GitHub Repository Deleted

[shashank-elastic]

These are min_stack version updates 🙂 Part of the PR
#4273

With the changing requirements of the integration versions, we identify the kibana version to minstack for rules. Where no query contents are changed in this case.

Will be presenting this into todays Simplified Protections Management Theme Working Group meeting

[shashank-elastic]

Notable Debugging and Root Cause Identification

• For the Rules listed above we have seen a major identifier that the related integration filed is not populated.

• For example GitHub Repository Deleted

  • No Related Integration Populated
    
    
  • We believe this is what causing the version diff to not show related integrations filed changes
    

• On rules where we have the related integration filed populated we are not seeing this issue

• For example GitHub Protected Branch Settings Changed

  • Related Integration Filed populated
    
  • Version changes show related integration version bump
    

• For these rules analysed The related integration is sporadically missing from the kibana asset.

  • Rule GitHub Repository Deleted does not have related integrations in kibana asset
    • https://github.com/elastic/integrations/blob/14991d9108f8e522ffbda85620900282d6a558bf/packages/security_detection_engine/kibana/security_rule/345889c4-23a8-4bc0-b7ca-756bd17ce83b_2.json#L13
    • https://github.com/elastic/integrations/blob/14991d9108f8e522ffbda85620900282d6a558bf/packages/security_detection_engine/kibana/security_rule/345889c4-23a8-4bc0-b7ca-756bd17ce83b_102.json#L13

• Rule GitHub Protected Branch Settings Changed has the related integrations in kibana asset

  • https://github.com/elastic/integrations/blob/14991d9108f8e522ffbda85620900282d6a558bf/packages/security_detection_engine/kibana/security_rule/07639887-da3a-4fbf-9532-8ce748ff8c50_4.json#L15
  • https://github.com/elastic/integrations/blob/14991d9108f8e522ffbda85620900282d6a558bf/packages/security_detection_engine/kibana/security_rule/07639887-da3a-4fbf-9532-8ce748ff8c50_105.json#L15

• The nature of this corrupted state is still unclear! because its the same code that generate assets for all rule types in the version!

• The reason I am stating that its very sporadic in nature is because another rule termed problematic in the above bug GitHub App Deleted is showing up rightly in my update diff

  • 
  • 
  • The asset also has related integration field populated https://github.com/elastic/integrations/blob/14991d9108f8e522ffbda85620900282d6a558bf/packages/security_detection_engine/kibana/security_rule/fd01b949-81be-46d5-bcf8-284395d5f56d_2.json#L16

cc @Mikaayenson @approksiu @banderror

[shashank-elastic]

Spot Check on My Test Stack

• Works as Expected Rules --> Meaning the Version Diff shows related integration Field, Kibana Asset has the related integration filed populated

Rule Name	Kibana Asset with related integrations
First Occurrence of Personal Access Token (PAT) Use For a GitHub User	Asset
High Number of Cloned GitHub Repos From PAT	Asset
First Occurrence of IP Address For GitHub Personal Access Token (PAT)	Asset
GitHub App Deleted	Asset
GitHub PAT Access Revoked	Asset
GitHub Owner Role Granted To User	Asset
First Occurrence of IP Address For GitHub User	Asset
GitHub User Blocked From Organization	Asset
First Occurrence of User-Agent For a GitHub User	Asset
First Occurrence GitHub Event for a Personal Access Token (PAT)	Asset
First Occurrence of GitHub User Interaction with Private Repo	Asset
First Occurrence of GitHub Repo Interaction From a New IP	Asset
GitHub Protected Branch Settings Changed	Asset
Member Removed From GitHub Organization	Asset
First Occurrence of User Agent For a GitHub Personal Access Token (PAT)	Asset
New GitHub App Installed	Asset
First Occurrence of Private Repo Event from Specific GitHub Personal Access Token (PAT)	Asset
New GitHub Owner Added	Asset
New User Added To GitHub Organization	Asset
GitHub Repo Created	Asset

• Bug --> As described above we dont have the related integration field populated.

Rule Name	Kibana Asset without related integrations
Multiple Okta User Authentication Events with Client Address	Asset
Multiple Okta User Authentication Events with Same Device Token Hash	Asset
Okta User Sessions Started from Different Geolocations	Asset
Multiple Device Token Hashes for Single Okta Session	Asset
High Number of Okta Device Token Cookies Generated for Authentication	Asset
GitHub Repository Deleted	Asset

Also out of the 26 rules reported by @banderror , I was able to reproduce only 6 of them :), the rest of them seem to render the right set of information for me
cc @Mikaayenson

[approksiu]

I see the same behavior as @banderror. @shashank-elastic I will share the instance where you can check. I see both issues - missing integration, and no changes in any rule fields. The rendering depends on the rule version you currently have, since you skipped some updates - there are more changes accumulated that you can view in diff.

[Mikaayenson]

@banderror @approksiu There are two issues.

1. Some rules do not have event.dataset when they should like GitHub Repository Deleted. This can be easily fixed.
2. Some rules are ES|QL. Since we do not have a parser for this language the build time fields are not populated. This includes related_integrations and required_fields.

[approksiu]

@Mikaayenson thanks! Are there plans to fix ES|QL rules build time field issues?

[shashank-elastic]

@Mikaayenson Yes thats true I was able to drill dow to the source of the First Version of the File that was added in release https://github.com/elastic/ia-trade-team/issues/388

The very first version of the rule, https://github.com/elastic/integrations/pull/10239/files#diff-0208dd40dfe6b85cb7f8bdba6bcd7305d84f9303bbf86514c8a0f9624157a8de does not have the related integrations_field.

To confirm the same here is another ESQL rule sample AWS STS Role Chaining that does not populate related integrations in Kibana

[shashank-elastic]

Have created a PR to fix even.dataset -#4278
cc @Mikaayenson @approksiu

[shashank-elastic]

@approksiu From your test stack I was able to reproduce the issue on the UI.

But from the asset perspective all the necessary required fields for related_integrations are rightly populated in non ES|QL supported rules.

Looking at the custom package that we created to test smart limits the only major difference we see is the json format

https://github.com/elastic/integrations/pull/11866/files#diff-49be033ff70d5dd333c4de4affe46c089228d1a26ec28fb57b4d8e0473f27724.

But all of the required information is rightly populated.

If we move to the latest version 8.16.2-beta.2 the file format is rectified https://github.com/elastic/integrations/pull/11900/files#diff-c0d42a1b673f8da2e46f126338934d5466bd7c9eb1f740f02870e4d3b2c5f238

But its unclear weather this is causing the issue, and since am not upgrading from a smart limit package, I may not be seeing this issue as in your stack or @banderror stack

From @approksiu Test Stack

From @shashank-elastic Test Stack

Also please not am moving from version 1--> 103 and the other stack is moving from 2 --> 103. One possible way to look at this is we are trying to diff from v8.16.2-beta.1 which was a custom package only to test smart limits, released ad-hoc.

So a conclusive debugging would be

• ES|QL Rule types don't have related_integrations filed populated, which makes the update diff only show version bumps and not actual changes
• Rule without event.dataset is fixed in PR Add event dataset for missing rule in Github integration #4278
• For other EQL rule types having empty diff based on the version bump movement we are seeing, it is likely that the previous version of the rule was from a custom built package to test only smart limits and we should be ideally looking at 8.16.1 --> 8.16.2 rule update diffs ( This is going by the description of the bug that we see this when we upgrade v8.16.2-beta.1 to v8.16.2)

Having said that, v8.16.2-beta.1 was temporary only a beta package to test smart limits and we have override that with the latest GA package so customers seeing this would be highly unlikely!

cc @Mikaayenson @banderror