[ti_opencti] Add support for IOC expiration

[chrisberkhout]

Proposed commit message

[ti_opencti] Add support for IOC expiration (#8920)

- Adds a transform to create an index of the latest non-expired and
  non-revoked indicators.
- A new field, `opencti.indicator.invalid_or_revoked_from` is added to
  combine information from `valid_until` and `revoked`/`modified_at`,
  for use in the retention policy.
- Field definition files are exactly the same for the source data
  stream and the transform, except for `ioc-transform-source.yml`,
  which is only present in the source data stream. For the ECS fields
  used, I added references to the external ECS field definitions, since
  this seems to have been necessary to get the right mappings into the
  transform destination index.
- Dashboards are updated. Previously existing dashboards now use the
  latest indicators only. A new Ingest dashboard shows both source and
  latest data.

Discussion

I'm assuming that opencti.indicator.valid_from will not have future values. That means I assume that an indicator will always be valid between the time its created and the time it expires or is revoked, rather than allowing e.g. "this will be valid starting next week".

As mentioned in the README, I don't clean up indicators that are never expired or revoked (a.k.a. "orphaned IOCs"), because I think that in the case of OpenCTI this can best be handled upstream or manually.

I started with the - external: ecs field definitions in a separate file from the existing ECS extensions and override definitions, but in the end I had to combine them into one file due to what seems to be a bug in Fleet.

I needed to raise the stack version in conditions.kibana.version from 8.9.0 to 8.10.1 to avoid this error:

Error: can't install the package: could not zip-install package; API status code
= 500; response body = {"statusCode":500,"error":"Internal Server Error","messag
e":"runtime_exception\n\tCaused by:\n\t\tillegal_argument_exception: unknown ind
ex sort field:[@timestamp]\n\tRoot causes:\n\t\truntime_exception: Could not cre
ate destination index [logs-ti_opencti_latest.dest_indicator-1] for transform [l
ogs-ti_opencti.latest_ioc-default-0.1.0]"}

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.

How to test this PR locally

You can test manually using the OpenCTI public demo instance. Note that the ingest dashboard will show immediate activity but it may take some time before non-revoked indicators are ingested and appear in the latest index.

The files shared between the data stream and the transform are the same:

diff -c9 <(cd data_stream/indicator/fields/              > /dev/null; md5sum *) \
         <(cd elasticsearch/transform/latest_ioc/fields/ > /dev/null; md5sum *)

*** /dev/fd/63	2024-01-17 18:24:53.214274662 +0100
--- /dev/fd/62	2024-01-17 18:24:53.214274662 +0100
***************
*** 1,4 ****
  3a08ed0a9ccffa200354ecc33f400c35  base-fields.yml
  b6ad99a97ee9877c06428b97a075ddeb  ecs.yml
- 5d8afa6ff186f571759469f8f0f575cc  ioc-transform-source.yml
  6204760fcfbcd8e1e4a264b656811306  opencti.yml
--- 1,3 ----

Related issues

• Relates Add IOC expiration support to all TI providers #5369 (meta issue)

Screenshots

Please see the updated screenshots (1-3) in the integration.

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

[chrisberkhout]

/test

[elasticmachine]

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

[efd6]

Minor nits only.

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 7aa3fe1

History

• 💔 Build #8779 failed 6c6bd7e5fdb4ed32b8c15734d6330155284c5422
• 💚 Build #8326 succeeded 0467f90
• 💚 Build #8310 succeeded ebfb734427ed5a1ace85a4b4d34599ee5697de31
• 💚 Build #8259 succeeded e3d535e99ba8442a9718a1c632034b0a97de2c50
• 💚 Build #8258 succeeded b0288dafc3b2b93f7bad9cbd1a29778a5a9c1d86

cc @chrisberkhout

[elastic-sonarqube]

Quality Gate passed

Kudos, no new issues were introduced!

0 New issues
0 Security Hotspots
100.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

[elasticmachine]

Package ti_opencti - 1.1.0 containing this change is available at https://epr.elastic.co/search?package=ti_opencti