Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] ListIOCsAPI total hits, and findings count per IOC are incorrect #1191

Open
AWSHurneyt opened this issue Jul 19, 2024 · 2 comments · May be fixed by #1373
Open

[BUG] ListIOCsAPI total hits, and findings count per IOC are incorrect #1191

AWSHurneyt opened this issue Jul 19, 2024 · 2 comments · May be fixed by #1373
Labels
bug Something isn't working

Comments

@AWSHurneyt
Copy link
Collaborator

What is the bug?
The "total" count returned by the ListIOCs API currently maxes at 10,000. This count should be the total number of IOCs that match the query.

The "numFindings" count for each IOC also currently maxes at 10,000. The query used to collect this count should ideally be refactored to an aggregation. Only the count of findings for each IOC needs to be returned.

@AWSHurneyt AWSHurneyt added bug Something isn't working untriaged and removed untriaged labels Jul 19, 2024
@AWSHurneyt
Copy link
Collaborator Author

The "total" count returned by the ListIOCs API currently maxes at 10,000. This count should be the total number of IOCs that match the query.

Started troubleshooting this locally by ingesting 10k IOCs in addition to the 609 IOCs that are ingested by the prepackaged AlienVault source.

GET localhost:9200/_cat/indices/.opensearch-sap-iocs-*?expand_wildcards=all&v
...
health status index                                                                uuid                   pri rep docs.count docs.deleted store.size pri.store.size
yellow open   .opensearch-sap-iocs-alienvault_reputation_ip_database-1729036845680 NLD2d9EmSN2XJFQVQyd6KA   1   1        609            0    253.7kb        253.7kb
yellow open   .opensearch-sap-iocs-uocgkpibvxtjl5g-zmls-1729036868686              GHLTDu7jSEGTYi7Z9CLMRg   1   1      10000            0      2.4mb          2.4mb

As called out above, the ListIOCs API incorrectly returns a maximum total of 10k.

GET localhost:9200/_plugins/_security_analytics/threat_intel/iocs?size=0
...
{
  "total": 10000,
  "iocs": []
}

However, even a general search query against the .opensearch-sap-iocs-* index pattern returns a total hit count of 10k.

GET localhost:9200/.opensearch-sap-iocs-*/_search
Content-Type: application/json
{
  "size": 0,
  "query": { "match_all": {} }
}
...
{
  "took": 17,
  "timed_out": false,
  "_shards": {
    "total": 2,
    "successful": 2,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 10000,
      "relation": "gte"
    },
    "max_score": null,
    "hits": []
  }
}

A _count API call shows the correct amount though.

GET localhost:9200/.opensearch-sap-iocs-*/_count
...
{
  "count": 10609,
  "_shards": {
    "total": 2,
    "successful": 2,
    "skipped": 0,
    "failed": 0
  }
}

@AWSHurneyt
Copy link
Collaborator Author

Regarding #1191 (comment), _search requests require track_total_hits to be set to true to return a hits.total.value greater than 10k.

I'll refactor this locally to fixed that portion of this bug report.

@AWSHurneyt AWSHurneyt linked a pull request Oct 23, 2024 that will close this issue
5 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant