[ML] Not for production - Add example Transform package with new specifications for testing of new installation mechanism #4138

qn895 · 2022-09-06T04:14:58Z

What does this PR do?

This PR is not intended for production. It simply showcases a new example package containing several transforms with specifications using the new specs elastic/package-spec#307 for testing purposes.

To get the package up and running in a test registry:
https://github.com/elastic/elastic-package

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

[ ]

How to test this PR locally

Related issues

Screenshots

elasticmachine · 2022-09-06T04:29:08Z

💔 Build Failed

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2022-09-15T21:19:55.950+0000
Duration: 12 min 46 sec

Steps errors

Expand to view the steps failures

`Test integration: transform`

Took 0 min 6 sec . View more details here
Description: eval "$(../../build/elastic-package stack shellinit)" ../../build/elastic-package test -v --report-format xUnit --report-output file --test-coverage

`Google Storage Download`

Took 0 min 0 sec . View more details here

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

szabosteve

Left a couple of minor comments to consider – mostly nits.

szabosteve · 2022-09-07T09:50:39Z

packages/transform/elasticsearch/transform/metadata_current/transform.yml

+  unique_key:
+    - elastic.agent.id
+  sort: "@timestamp"
+description: Latest Endpoint metadata document per host


Suggested change

description: Latest Endpoint metadata document per host

description: Latest endpoint metadata document per host.

Updated here 6f89697

szabosteve · 2022-09-07T10:00:05Z

packages/transform/elasticsearch/transform/metadata_united/transform.yml

+    agent.id:
+      terms:
+        field: agent.id
+description: Merges latest Endpoint and Agent metadata documents


Suggested change

description: Merges latest Endpoint and Agent metadata documents

description: Merges latest endpoint and Agent metadata documents

Updated here 6f89697

szabosteve · 2022-09-07T10:52:39Z

packages/transform/manifest.yml

+source:
+  license: "Elastic-2.0"
+license: basic
+description: "Installs and optionally starts multiple example transform tasks"


If it's the description to use on one of the tiles on the Browse integrations page. Then I suggest the following changes so the description will be in line with the rest of the integration descriptions. (Even though the descriptions are not entirely consistent throughout the page.)

If it'll be used somewhere else, please ignore my comment.

Suggested change

description: "Installs and optionally starts multiple example transform tasks"

description: "Install and optionally start multiple example transform tasks."

Updated here 6f89697

droberts195 · 2022-09-08T15:41:15Z

packages/transform/elasticsearch/transform/metadata_current/transform.yml

+      "@timestamp":
+        gt: now-90d/d
+dest:
+  index: metrics-endpoint.metadata_current_default


It's interesting that default is hardcoded here. While running through installation end-to-end I noticed that the Integrations UI lets you customise the "namespace" for the package. The "namespace" is "default" by default. But presumably if you change it then it changes the names of some of the assets installed by the package. So maybe we should change default here in that case as well, to match the chosen namespace.

In general (not just for transforms) it might not be very well thought through how "namespace" influences what gets created when a package is installed.

As a companion to elastic/kibana#140046 an example transform package is being created in elastic/integrations#4138. Initially the intention was that this example would use the Kibana sample data. Then, after realising that wouldn't work due to lack of permissions, it was changed to reinstall a transform that is created by the security solution. However, it has been pointed out that this is dangerous because even if we tell people not to install the example in production it could somehow happen by mistake and then we'd have created objects that clash with production code. Therefore, it is best that the example transform package be switched back to the original idea of using the Kibana sample data, so that there is no risk of it ever clashing with a production package.

…w-example-package

qn895 · 2022-09-16T18:47:10Z

@droberts195 I've updated this package to use the Kibana sample ecommerce dataset now that elastic/elasticsearch#90037 is merged. The examples include a pivot and a latest transform.

droberts195 · 2022-09-20T13:56:33Z

I've updated this package to use the Kibana sample ecommerce dataset

Thanks @qn895. The idea of using the Kibana sample ecommerce dataset and having separate pivot and latest transforms looks good to me. The details of the schema might need changing based on comments others are adding to elastic/kibana#140046. But I think the overall approach of creating an example that cannot possibly interfere with production transforms if end users install it is correct.

botelastic · 2022-10-20T14:02:22Z

Hi! We just realized that we haven't looked into this PR in a while. We're sorry! We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!

botelastic · 2022-11-19T15:01:55Z

Hi! This PR has been stale for a while and we're going to close it as part of our cleanup procedure. We appreciate your contribution and would like to apologize if we have not been able to review it, due to the current heavy load of the team. Feel free to re-open this PR if you think it should stay open and is worth rebasing. Thank you for your contribution!

qn895 added 2 commits September 5, 2022 22:59

[ML] Add example transform packages according to new specs

636096e

Update package version, owner

5e9e1af

qn895 changed the title ~~[ML] Add example transform packages according to new specs~~ [ML] Add example Transform package according to new specs Sep 6, 2022

Update changelog.yml

548ad12

qn895 mentioned this pull request Sep 6, 2022

[ML] Update installation mechanism for Transforms in Fleet according to new specifications elastic/kibana#140046

Merged

19 tasks

qn895 added 3 commits September 6, 2022 10:05

[ML] Rename module

54a3354

[ML] Delete ecommerce ex because we don't have permission to delete

9cd5e5e

[ML] Don't start one of the transform

a147689

szabosteve reviewed Sep 7, 2022

View reviewed changes

droberts195 reviewed Sep 8, 2022

View reviewed changes

[ML] Update new descriptions

6f89697

qn895 changed the title ~~[ML] Add example Transform package according to new specs~~ [ML] Not for production - Add example Transform package with new specifications for testing new installation mechanism Sep 12, 2022

qn895 changed the title ~~[ML] Not for production - Add example Transform package with new specifications for testing new installation mechanism~~ [ML] Not for production - Add example Transform package with new specifications for testing of new installation mechanism Sep 12, 2022

droberts195 mentioned this pull request Sep 13, 2022

Add privileges for Fleet example package for transforms elastic/elasticsearch#90037

Merged

qn895 added 3 commits September 15, 2022 16:17

[ML] Update package to use ecommerce examples

b88a578

Merge remote-tracking branch 'upstream/main' into ml-transform-add-ne…

f0834d1

…w-example-package

[ML] Update package to use ecommerce examples

2ff2793

qn895 force-pushed the ml-transform-add-new-example-package branch from c53e261 to 2ff2793 Compare September 15, 2022 21:19

qn895 self-assigned this Sep 16, 2022

botelastic bot added the Stalled label Oct 20, 2022

botelastic bot closed this Nov 19, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[ML] Not for production - Add example Transform package with new specifications for testing of new installation mechanism #4138

[ML] Not for production - Add example Transform package with new specifications for testing of new installation mechanism #4138

Uh oh!

qn895 commented Sep 6, 2022 •

edited

Loading

Uh oh!

elasticmachine commented Sep 6, 2022 •

edited

Loading

Build stats

`Test integration: transform`

`Google Storage Download`

Uh oh!

szabosteve left a comment

Uh oh!

szabosteve Sep 7, 2022

Uh oh!

qn895 Sep 8, 2022

Uh oh!

szabosteve Sep 7, 2022

Uh oh!

qn895 Sep 8, 2022

Uh oh!

szabosteve Sep 7, 2022

Uh oh!

qn895 Sep 8, 2022

Uh oh!

droberts195 Sep 8, 2022

Uh oh!

qn895 commented Sep 16, 2022

Uh oh!

droberts195 commented Sep 20, 2022

Uh oh!

botelastic bot commented Oct 20, 2022

Uh oh!

botelastic bot commented Nov 19, 2022

Uh oh!

Uh oh!

	description: Latest Endpoint metadata document per host
	description: Latest endpoint metadata document per host.

	description: Merges latest Endpoint and Agent metadata documents
	description: Merges latest endpoint and Agent metadata documents

	description: "Installs and optionally starts multiple example transform tasks"
	description: "Install and optionally start multiple example transform tasks."

[ML] Not for production - Add example Transform package with new specifications for testing of new installation mechanism #4138

[ML] Not for production - Add example Transform package with new specifications for testing of new installation mechanism #4138

Uh oh!

Conversation

qn895 commented Sep 6, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Checklist

Author's Checklist

How to test this PR locally

Related issues

Screenshots

Uh oh!

elasticmachine commented Sep 6, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💔 Build Failed

Build stats

Steps errors

Test integration: transform

Google Storage Download

🤖 GitHub comments

Uh oh!

szabosteve left a comment

Choose a reason for hiding this comment

Uh oh!

szabosteve Sep 7, 2022

Choose a reason for hiding this comment

Uh oh!

qn895 Sep 8, 2022

Choose a reason for hiding this comment

Uh oh!

szabosteve Sep 7, 2022

Choose a reason for hiding this comment

Uh oh!

qn895 Sep 8, 2022

Choose a reason for hiding this comment

Uh oh!

szabosteve Sep 7, 2022

Choose a reason for hiding this comment

Uh oh!

qn895 Sep 8, 2022

Choose a reason for hiding this comment

Uh oh!

droberts195 Sep 8, 2022

Choose a reason for hiding this comment

Uh oh!

qn895 commented Sep 16, 2022

Uh oh!

droberts195 commented Sep 20, 2022

Uh oh!

botelastic bot commented Oct 20, 2022

Uh oh!

botelastic bot commented Nov 19, 2022

Uh oh!

Uh oh!

qn895 commented Sep 6, 2022 •

edited

Loading

elasticmachine commented Sep 6, 2022 •

edited

Loading

`Test integration: transform`

`Google Storage Download`