Security ai prompts #871

patrykkopycinski · 2025-02-11T10:34:21Z

Adds kibana/security_ai_prompt to support security AI prompt assets.

Why is it important?

In order to have the flexibility to tweak AI prompts outside of our regular ESS release schedule, kibana/security_ai_prompt assets introduce the ability to ship prompt updates for the security AI Assistant and Attack Discovery.

Checklist

I have added test packages to test/packages that prove my change is effective.
I have added an entry in spec/changelog.yml.

Related issues

https://github.com/elastic/security-team/issues/11196

Adds `kibana/security_ai_prompt` to support security AI prompt assets. ## Why is it important? In order to have the flexibility to tweak AI prompts outside of our regular ESS release schedule, `kibana/security_ai_prompt` assets introduce the ability to ship prompt updates for the security AI Assistant and Attack Discovery. ## Checklist - [x] I have added test packages to [`test/packages`](https://github.com/elastic/package-spec/tree/main/test/packages) that prove my change is effective. - [x] I have added an entry in [`spec/changelog.yml`](https://github.com/elastic/package-spec/blob/main/spec/changelog.yml). ## Related issues - <elastic/security-team#11196>

patrykkopycinski · 2025-02-11T10:34:27Z

test integrations

elastic-vault-github-plugin-prod · 2025-02-11T10:43:46Z

Created or updated PR in integrations repository to test this version. Check elastic/integrations#12719

jsoriano · 2025-02-11T12:11:46Z

Duplicate of #870? I was commenting on the other one 😅

patrykkopycinski · 2025-02-11T12:27:03Z

yeah, I just want to test it before @andrew-goldstein wakes up :)

spec/integration/kibana/spec.yml

test/packages/good_v2/kibana/security_ai_prompt/good_v2-security-ai-prompt-1.json

mrodm · 2025-03-26T17:38:06Z

spec/integration/kibana/spec.yml

+  - description: Folder containing security AI prompt assets
+    type: folder
+    name: "security_ai_prompt"
+    required: false
+    contents:
+    - description: A security AI prompt asset file
+      type: file
+      contentMediaType: "application/json"
+      pattern: '^{PACKAGE_NAME}-.+\.json$'


Would it make sense to add this Kibana asset to content packages? Or is it intended just for integration packages ?

I’m not sure what "content packages" is?

Reading through the the Package specification section of the Integrations Developer Guide, I don't believe this package should be a "content package" since the assets do not include any dashboards, visualizations, or ingest pipelines to enhance the user experience. The assets are JSON saved objects to deliver text at certain points in AI workflows. The user does not see the additional prompting, it is sent to the LLM.

Or do you mean defining it in the Kibana repo? This is already done: https://github.com/elastic/kibana/blob/main/x-pack/platform/plugins/shared/fleet/common/types/models/epm.ts#L62

Looking at the integrations PR elastic/integrations#13323 you linked, this looks like a candidate to be also available in content packages (under Kibana folder too with the JSON path to remove it in previous versions). Integration packages allow to define data streams, mappings, ingest pipelines and it looks like that all those resources would not be needed for these security AI prompt assets to work (is that right?).

I think it could also be kept as part of integration packages, in case it also needed to add/provide Security AI prompts in integration packages.

WDYT @jsoriano ?

If you have branches of integrations or elastic-package using this change you will need to update the references, yes.

The docs say that elastic-package supports only the "integration" package type. Is that right? How do I update the references?

The docs say that elastic-package supports only the "integration" package type. Is that right?

This is probably outdated, elastic-package supports integration, input and content packages. Where have you seen this?

How do I update the references?

These lines should do the trick in the integrations or elastic-package repos:

go mod edit -replace github.com/elastic/package-spec/v3=github.com/patrykkopycinski/package-spec/v3@security_ai_prompts go mod tidy

elastic-package supports integration, input and content packages. Where have you seen this?

Thank you. Here is the outdated doc reference https://www.elastic.co/guide/en/integrations-developer/current/elastic-package.html#_elastic_package_build

Updating docs elastic/elastic-package#2507

stephmilovic · 2025-03-27T14:36:03Z

question for @mrodm / @jsoriano
Once this is merged, how do I update the elastic-package repo? Does this happen through a bot or is there something I need to do? Then once elastic-package is updated, I think I'll need to update the go.mod and go.sum in the integrations repo. Is there a script I can run to do this, or is it a manual step? Thanks

mrodm · 2025-03-27T16:01:42Z

question for @mrodm / @jsoriano Once this is merged, how do I update the elastic-package repo? Does this happen through a bot or is there something I need to do? Then once elastic-package is updated, I think I'll need to update the go.mod and go.sum in the integrations repo. Is there a script I can run to do this, or is it a manual step? Thanks

To release these changes and make them available up to the integrations repository, there are a few steps that needs to be performed:

release a new package-spec version.
update the package-spec dependency into elastic-package and release a new version of elastic-package.
update the elastic-package dependency into integrations repo.

However, the version of the spec where these new assets are available would depend on when the support in Fleet/Kibana is added too.

stephmilovic · 2025-03-27T16:46:16Z

support in Fleet/Kibana is added too

Support is already in Kibana/Fleet. However, it would be a manual installation. This PR is to automatically install the integration and registers the package version, and it will target 8.19.0/9.1.0 elastic/kibana#216106

jsoriano · 2025-03-27T20:50:43Z

support in Fleet/Kibana is added too

Support is already in Kibana/Fleet. However, it would be a manual installation. This PR is to automatically install the integration and registers the package version, and it will target 8.19.0/9.1.0 elastic/kibana#216106

When talking about support in Fleet we refer to Fleet being able to install these assets when found in a package.

If we are targeting unreleased versions of Kibana we should use a new minor, 3.4.0 sounds good.

stephmilovic · 2025-03-31T14:42:05Z

anything else I need on this PR for approval @jsoriano ? Thanks!

jsoriano

In principle the change LGTM. I would still like to see a test package with a more complete use case.

jsoriano · 2025-04-01T15:02:27Z

test/packages/good_content/kibana/security_ai_prompt/good_content-security-ai-prompt-1.json

+  },
+  "id": "good_content-security-ai-prompt-1",
+  "type": "security-ai-prompt"
+}


I am still missing a sample with a real use case. Is the idea to distribute a package with a collection of prompts? Or to associate prompts to specific data?

Sorry, I'm confused what else you need here? All the package does is distribute prompts as saved objects. The prompts are looked up with the saved object API within security solution. If no saved object prompt exists, a fallback prompt value exists. https://github.com/elastic/kibana/blob/0d415a6d3a09200dad48a58851d89d81ef897b81/x-pack/solutions/security/packages/security-ai-prompts/src/get_prompt.ts#L109-L113

I am not sure about the package that is planned after this new asset is supported. Will it be a package that will contain only a collection of prompts? Or will prompts be added to other packages?

There will be one new package that contains only a collection of prompts as saved objects, see here: https://github.com/elastic/integrations/pull/13323/files

Ok, then the model proposed here based on content package will fit great.

elasticmachine · 2025-04-02T09:15:41Z

💚 Build Succeeded

Buildkite Build
Commit: b907186

History

💚 Build #1076 succeeded ad2c8bf
💔 Build #1075 failed e54eb29
💔 Build #1074 failed 9551e8f
💔 Build #1073 failed 5945ba6
💚 Build #1071 succeeded 77dacdf
💔 Build #1070 failed 5927910

andrew-goldstein and others added 2 commits February 10, 2025 09:47

fix tests

45b6984

elastic-vault-github-plugin-prod bot mentioned this pull request Feb 11, 2025

Test branch package-spec#871 - DO NOT MERGE elastic/integrations#12719

Closed

stephmilovic mentioned this pull request Mar 25, 2025

## What does this PR do? #870

Closed

4 tasks

stephmilovic added 2 commits March 26, 2025 08:17

updates from 870

78cf47a

Merge branch 'main' into security_ai_prompts

2d284ba

jsoriano reviewed Mar 26, 2025

View reviewed changes

spec/integration/kibana/spec.yml Outdated Show resolved Hide resolved

fix spec/integration/kibana/spec.yml

5927910

stephmilovic reviewed Mar 26, 2025

View reviewed changes

test/packages/good_v2/kibana/security_ai_prompt/good_v2-security-ai-prompt-1.json Outdated Show resolved Hide resolved

rm sample json from good and good_v2

77dacdf

stephmilovic marked this pull request as ready for review March 26, 2025 17:30

stephmilovic requested a review from a team as a code owner March 26, 2025 17:30

mrodm reviewed Mar 26, 2025

View reviewed changes

This was referenced Mar 26, 2025

[Security Solution] Security AI Prompts elastic/integrations#13323

Draft

[Security solution] Security AI prompts integration elastic/kibana#216106

Draft

integration => content

5945ba6

stephmilovic added 3 commits March 27, 2025 14:52

add space back

9551e8f

move test

e54eb29

fix test

ad2c8bf

jsoriano previously approved these changes Apr 1, 2025

View reviewed changes

Merge branch 'main' into security_ai_prompts

b907186

jsoriano dismissed their stale review via b907186 April 2, 2025 09:00

jsoriano enabled auto-merge (squash) April 2, 2025 09:01

jsoriano approved these changes Apr 2, 2025

View reviewed changes

jsoriano merged commit ed8dc58 into elastic:main Apr 2, 2025
3 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Security ai prompts #871

Security ai prompts #871

patrykkopycinski commented Feb 11, 2025 •

edited by stephmilovic

Loading

patrykkopycinski commented Feb 11, 2025

elastic-vault-github-plugin-prod bot commented Feb 11, 2025

jsoriano commented Feb 11, 2025

patrykkopycinski commented Feb 11, 2025

mrodm Mar 26, 2025

stephmilovic Mar 27, 2025

stephmilovic Mar 27, 2025

stephmilovic Mar 27, 2025

mrodm Mar 27, 2025

jsoriano Mar 27, 2025

stephmilovic Mar 27, 2025

jsoriano Mar 27, 2025

stephmilovic Mar 27, 2025

jsoriano Apr 1, 2025

stephmilovic commented Mar 27, 2025

mrodm commented Mar 27, 2025

stephmilovic commented Mar 27, 2025 •

edited

Loading

jsoriano commented Mar 27, 2025

stephmilovic commented Mar 31, 2025

jsoriano left a comment

jsoriano Apr 1, 2025

stephmilovic Apr 1, 2025

jsoriano Apr 1, 2025

stephmilovic Apr 1, 2025

jsoriano Apr 2, 2025

elasticmachine commented Apr 2, 2025

Security ai prompts #871

Security ai prompts #871

Conversation

patrykkopycinski commented Feb 11, 2025 • edited by stephmilovic Loading

Why is it important?

Checklist

Related issues

patrykkopycinski commented Feb 11, 2025

elastic-vault-github-plugin-prod bot commented Feb 11, 2025

jsoriano commented Feb 11, 2025

patrykkopycinski commented Feb 11, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stephmilovic commented Mar 27, 2025

mrodm commented Mar 27, 2025

stephmilovic commented Mar 27, 2025 • edited Loading

jsoriano commented Mar 27, 2025

stephmilovic commented Mar 31, 2025

jsoriano left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

elasticmachine commented Apr 2, 2025

💚 Build Succeeded

History

patrykkopycinski commented Feb 11, 2025 •

edited by stephmilovic

Loading

stephmilovic commented Mar 27, 2025 •

edited

Loading