[ML] Adding space aware jobs #77916

jgowdyelastic · 2020-09-18T16:02:37Z

Adds saved objects to represent ML anomaly detection and data frame analytics jobs to allow for them to be assigned to spaces.

Related to #64172

The new saved object has the type ml-job, and is this shape:

ml-job: {
  job_id: "foo"
  datafeed_id: "datafeed-foo",
  type: "anomaly-detector"
},

datafeed_id will be null for dataframe analytics jobs and anomaly detector jobs with no datafeed.
type will be either anomaly-detector or data-frame-analytics

Job id checks and filters have been added to all appropriate kibana ml endpoints.
For example:

GET <kibana>/abc/api/ml/anomaly_detectors

will only return jobs in the current space.

GET <kibana>/abc/api/ml/anomaly_detectors/foo

Will return a 404 if foo is not in the current space.

All anomaly results searches now use a new function anomalySearch which also requires the list a job ids.
If any of the job ids listed does not exist or is not in the current space, a 404 will be returned.

Things to note when testing:

When creating a job in the UI, the job will only exist in the current space.
- Future work (7.11) will add the ability to move jobs between spaces in kibana's management app.
When deleting a job, it will be deleted for all spaces.
- Future work (7.11) will instead remove the job from the current space and only delete the actual job when it is assigned to only one space.
When using the assign_job_to_space or remove_job_from_space endpoints, the job must exist in the current space.
- This will be fixed in a follow up PR.

When testing this PR, any pre-existing jobs will not have saved objects assigned to them and so will not appear in the UI.
These saved objects can be created by calling:

GET <kibana>/api/ml/saved_objects/repair

To move jobs between spaces:

POST <kibana>/api/ml/saved_objects/assign_job_to_space
or
POST <kibana>/api/ml/saved_objects/remove_job_from_space

Payload:
{
  "jobType": "anomaly-detector",
  "jobIds": [<jobIds>],
  "spaces": [<spaces names>]
}

jobType can be anomaly-detector and data-frame-analytics

Job spaces are also listed in the management page:

For APM reviewers, I had to recreate the apm_8.0.0 archive to add the ML saved objects to the snapshot .kibana index.

Follow up work for 7.11:

Job management UI for assigning spaces
Allow assigning of jobs to any space accessible by the user.
Manage job deletion, only remove job from space if it is assigned to multiple spaces.
ES upgrade, migration task should create saved objects in * space for all existing jobs.
Add functional API tests. We should have strong coverage for all endpoints modified.

Checklist

Delete any items that are not applicable to this PR.

Documentation was added for features that require explanation or tutorials
Unit or functional tests were updated or added to match the most common scenarios
This was checked for breaking API changes and was labeled appropriately

jgowdyelastic · 2020-09-28T08:01:18Z

@elasticmachine merge upstream

jgowdyelastic · 2020-09-28T08:05:10Z

@elasticmachine merge upstream

jgowdyelastic · 2020-09-28T08:40:47Z

@elasticmachine merge upstream

jgowdyelastic · 2020-09-29T10:27:07Z

@elasticmachine merge upstream

jgowdyelastic · 2020-09-30T13:37:51Z

@elasticmachine merge upstream

jgowdyelastic · 2020-10-01T11:46:41Z

@elasticmachine merge upstream

…bana into adding-space-aware-jobs

jgowdyelastic · 2020-10-01T13:35:52Z

@elasticmachine merge upstream

…bana into adding-space-aware-jobs

jgowdyelastic · 2020-10-05T07:58:25Z

@elasticmachine merge upstream

alvarezmelissa87

Latest changes LGTM ⚡

peteharverson · 2020-10-30T13:13:49Z

If I disable Spaces, with xpack.spaces.enabled: false, I only see jobs from the default space. Is this the correct behavior?
cc @legrego

peteharverson

Left a question about the behaviour when spaces is disabled, but otherwise tested latest edits and LGTM.

jgowdyelastic · 2020-10-30T13:34:06Z

If I disable Spaces, with xpack.spaces.enabled: false, I only see jobs from the default space. Is this the correct behavior?
cc @legrego

I'm not doing anything special here if spaces is disabled. i'm still using the saved object client and these are the jobs it's returning.
I assumed this was intended behaviour.
We would want to continue creating and searching the saved objects when creating and retrieving jobs, just in case spaces is enabled at a future point.

legrego · 2020-10-30T14:04:01Z

If I disable Spaces, with xpack.spaces.enabled: false, I only see jobs from the default space. Is this the correct behavior?
cc @legrego

I'm not doing anything special here if spaces is disabled. i'm still using the saved object client and these are the jobs it's returning.
I assumed this was intended behaviour.
We would want to continue creating and searching the saved objects when creating and retrieving jobs, just in case spaces is enabled at a future point.

Yes this is intended behavior, and consistent with the way other saved objects behave today. It's not the best experience, but that's what we have.

legrego · 2020-11-01T17:38:12Z

My plan was to finish my review on Monday, but I'm taking a sick day instead; I'll pick this back up as soon as possible.

jgowdyelastic · 2020-11-02T09:32:16Z

@elasticmachine merge upstream

darnautov

LGTM

walterra

Code LGTM 🥳

jgowdyelastic · 2020-11-02T16:34:39Z

@elasticmachine merge upstream

jgowdyelastic · 2020-11-03T12:45:25Z

@elasticmachine merge upstream

jgowdyelastic · 2020-11-03T13:44:16Z

@elasticmachine merge upstream

kibanamachine · 2020-11-03T15:30:07Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: 563526a

Metrics [docs]

@kbn/optimizer bundle module count

id	before	after	diff
`ml`	1234	1237	+3

async chunks size

id	before	after	diff
`ml`	6.6MB	6.6MB	+1.9KB

distributable file count

id	before	after	diff
`default`	42620	42633	+13

page load bundle size

id	before	after	diff
`ml`	63.5KB	63.5KB	-41.0B

Saved Objects .kibana field count

id	before	after	diff
`ml-job`	-	6	+6

History

💚 Build #85490 succeeded 35a981a
💚 Build #85363 succeeded 13ccded
💔 Build #85269 failed 660af5c
💔 Build #85260 failed 9cfafac
💚 Build #85071 succeeded cd07a6d

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

* [ML] Adding space aware jobs * adding mlClient * switching to type includes * adding additional job checks * fixing conflict * adding dfa checks * refactoring jobs in spaces checks * filtering calendars * adding initial job object status and repair endpoints * enabling repair endpoint * fixing listed jobs in status * adding datafeed repair * updating shared services * adding results job id check * fixing conflicts * don't remove SO on delete * fixing non-ml plugins * filtering job audit messages * fixing types * fixing tests * adding job ids wildcard support * removing empty migration test * fixing tests and disabling spaces test user * adding saved objects all permission * fixing calendars * updating job 404 * updating job wildcard search * renaming services * fixing conflicts * fixing log tests * disabling apm test * skipping more apm tests * optimzing repair * fixing types * updating apm test archive to include ML saved objects * enabling disabled test * removing comment * adding space assigning endpoints * adding saved object default permissions * removing commented code * loading all jobs for all spaces for status check * adding spaces list endpoint * adding job spaces to management page * adding trained model fltering * fixing trained model id check and job wildcard check * fixing types * fixing bug when adding new job to calendar * changes based on review * updating schema * changes based on review * fixing types * rolling back http service injection * fixing http service injection * adding errrors to repair endpoint response * updating api doc * improving types * disabling id check on ad get endpoints * fixing tests * fixing group requests * adding comments * using filter in saved object search * fixing fake request issue * removing console log * making job saved object hidden * removing acccidentally included file * renaming saved object client * updating apidoc * unhiding ml saved objects * moving route guard * improving error when SOC is null * fixing types after merge with master * fixing tests Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com> Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com> Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

[ML] Adding space aware jobs

ba7bdae

joshdover mentioned this pull request Sep 22, 2020

[Meta] Saved Object Tagging #74571

Closed

41 tasks

Merge branch 'master' into adding-space-aware-jobs

4a8d36a

elasticmachine and others added 3 commits September 29, 2020 06:27

Merge branch 'master' into adding-space-aware-jobs

7793ced

adding mlClient

3397246

switching to type includes

837ad42

elasticmachine and others added 4 commits September 30, 2020 09:37

Merge branch 'master' into adding-space-aware-jobs

15a04e8

adding additional job checks

f430b02

Merge branch 'master' into adding-space-aware-jobs

5ba9d98

fixing conflict

f049540

elasticmachine and others added 2 commits October 1, 2020 07:46

Merge branch 'master' into adding-space-aware-jobs

cd76629

Merge branch 'adding-space-aware-jobs' of github.com:jgowdyelastic/ki…

815219c

…bana into adding-space-aware-jobs

elasticmachine and others added 5 commits October 1, 2020 09:35

Merge branch 'master' into adding-space-aware-jobs

837fac0

Merge branch 'adding-space-aware-jobs' of github.com:jgowdyelastic/ki…

6b2ff4c

…bana into adding-space-aware-jobs

adding dfa checks

2e4674b

refactoring jobs in spaces checks

0490d31

filtering calendars

a6fcb33

kibanamachine and others added 5 commits October 5, 2020 03:58

Merge branch 'master' into adding-space-aware-jobs

4a33827

adding initial job object status and repair endpoints

b956b83

enabling repair endpoint

9881370

fixing listed jobs in status

72661e8

adding datafeed repair

5495f6b

alvarezmelissa87 approved these changes Oct 29, 2020

View reviewed changes

peteharverson approved these changes Oct 30, 2020

View reviewed changes

Merge branch 'master' into adding-space-aware-jobs

cd07a6d

darnautov approved these changes Nov 2, 2020

View reviewed changes

walterra approved these changes Nov 2, 2020

View reviewed changes

Merge branch 'master' into adding-space-aware-jobs

3ba5c0e

kibanamachine and others added 3 commits November 2, 2020 11:34

Merge branch 'master' into adding-space-aware-jobs

9cfafac

fixing types after merge with master

660af5c

fixing tests

13ccded

jgowdyelastic mentioned this pull request Nov 3, 2020

[ML] [Meta] Spaces follow up tasks #82415

Closed

13 tasks

Merge branch 'master' into adding-space-aware-jobs

35a981a

Merge branch 'master' into adding-space-aware-jobs

563526a

jgowdyelastic merged commit a0fded5 into elastic:master Nov 3, 2020

jgowdyelastic mentioned this pull request Nov 3, 2020

[7.x] [ML] Adding space aware jobs (#77916) #82446

Merged

jgowdyelastic deleted the adding-space-aware-jobs branch November 3, 2020 17:39

spong mentioned this pull request Nov 4, 2020

[Security Solution][Detections] Ensure ML Detection Rules/Jobs function with introduction of Space Aware ML Jobs #82553

Closed

weltenwort mentioned this pull request Nov 18, 2020

[Logs UI] Migrate to the new space-aware ML APIs #67936

Closed

pgayvallet mentioned this pull request Dec 14, 2020

[meta] saved object tagging: integration in Kibana applications #85741

Open

12 tasks

peteharverson mentioned this pull request Dec 22, 2020

[ML] Integration with Kibana Spaces #64172

Closed

26 tasks

jgowdyelastic mentioned this pull request Jun 18, 2021

[ML] Calendars and Filters in spaces #102657

Open

[ML] Adding space aware jobs #77916

[ML] Adding space aware jobs #77916

Uh oh!

Conversation

jgowdyelastic commented Sep 18, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Checklist

Uh oh!

jgowdyelastic commented Sep 28, 2020

Uh oh!

jgowdyelastic commented Sep 28, 2020

Uh oh!

jgowdyelastic commented Sep 28, 2020

Uh oh!

jgowdyelastic commented Sep 29, 2020

Uh oh!

jgowdyelastic commented Sep 30, 2020

Uh oh!

jgowdyelastic commented Oct 1, 2020

Uh oh!

jgowdyelastic commented Oct 1, 2020

Uh oh!

jgowdyelastic commented Oct 5, 2020

Uh oh!

alvarezmelissa87 left a comment

Choose a reason for hiding this comment

Uh oh!

peteharverson commented Oct 30, 2020

Uh oh!

peteharverson left a comment

Choose a reason for hiding this comment

Uh oh!

jgowdyelastic commented Oct 30, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

legrego commented Oct 30, 2020

Uh oh!

legrego commented Nov 1, 2020

Uh oh!

jgowdyelastic commented Nov 2, 2020

Uh oh!

darnautov left a comment

Choose a reason for hiding this comment

Uh oh!

walterra left a comment

Choose a reason for hiding this comment

Uh oh!

jgowdyelastic commented Nov 2, 2020

Uh oh!

jgowdyelastic commented Nov 3, 2020

Uh oh!

jgowdyelastic commented Nov 3, 2020

Uh oh!

kibanamachine commented Nov 3, 2020

💚 Build Succeeded

Metrics [docs]

@kbn/optimizer bundle module count

async chunks size

distributable file count

page load bundle size

Saved Objects .kibana field count

History

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

15 participants

jgowdyelastic commented Sep 18, 2020 •

edited

Loading

jgowdyelastic commented Oct 30, 2020 •

edited

Loading