[Ingest Manager] Rename data sources to package configs #70259

jen-huang · 2020-06-29T20:53:31Z

Summary

Resolves #70018 as part of elastic/beats#19082. This PR replaces datasource(s)/data source(s) with package config(s) everywhere in Ingest Manager and Endpoint code. This PR has no visible impact apart from small UI copy changes and routing path changes. The affected areas include:

Saved object type ingest-datasources -> ingest-package-config
- Field name agent_stream -> compiled_stream
- Removed unused processors field
Field datasources on agent config SO renamed to package_configs
Removed all 7.9 Ingest Manager SO migrations as we are not supporting an upgrade path from experimental to beta release
Ingest manager API routes starting with /api/ingest_manager/datasources -> /api/ingest_manager/package_configs
Ingest manager app routes /app/ingestManager/.../(add|edit)-datasource -> /app/ingestManager/.../(add|edit)-integration
Variables that use string d/Datasource(s) -> p/PackageConfig(s)
- Types, schemas, rest spec
- Components, services
File names that include datasource(s) -> package_config(s)
Code comments, dev docs
UI copy adjustments (partially resolves [Ingest Manager] Name changes tracking issue #70024)

Testing

Start with wiped ES data
Everything should function normally

Screenshots

Showing revised copy, breadcrumbs, and URL path:

…-datasource` to `ingest-package-config`) and adjust mappings: - Remove unused `processors` field on input & stream levels - Remove unnecessary `enabled` field on package config & input levels - Rename `agent_stream` field to `compiled_stream` - Reorder other fields so that important fields are closer to top & similar fields are grouped together Also, - Remove 7.9.0 migrations as we are not supporting an upgrade path from experimental to beta release

…rences (does not include actual type definition changes yet)

…(does not include actual schema changes yet)

…configs` and update all references

… needed for current & future UI features. Also: - Match types and schemas with saved object mappings (`agent_stream` to `compiled_stream`, removal of `processors`) - Set `namespace` to be a required property on agent config and package config types, add validation support for it on UI

…s, and routes

…n client sides, and for Endpoint too

…ngest/policies` es archiver data

elasticmachine · 2020-06-29T23:28:52Z

Pinging @elastic/ingest-management (Team:Ingest Management)

jen-huang · 2020-06-30T00:08:43Z

x-pack/plugins/ingest_manager/server/routes/agent_config/handlers.ts

+    // Create the system monitoring package config and add it to agent config.
+    if (withSysMonitoring && newSysPackageConfig !== undefined && agentConfig !== undefined) {
+      newSysPackageConfig.config_id = agentConfig.id;
+      newSysPackageConfig.namespace = agentConfig.namespace;


This line fixes #68041

…asources

jen-huang · 2020-06-30T21:39:28Z

@elastic/ingest-management @elastic/endpoint-management

Hi teams, could one person from each team help look at this PR? It's probably easiest to review by pulling down locally (with wiped ES data) and trying it out, there should be no difference in functionality. It would also be great if you can spot check some files including a few test files.

Since so many files are affected, it'd be great to get this PR reviewed & merged sooner rather than later, so that we can minimize the churn of resolving conflicts.

…asources

kevinlog · 2020-07-01T15:51:49Z

Tried it out with our Policy flows - looks good!

kevinlog

This looks good from the Endpoint Management side, pulled it down and tested it - our flows still work and the steps still makes sense

jfsiii

Checked it out locally and clicked around. I can't test installing integrations because of #70453 or without a local/older registry.

I'm fine merging this and dealing with any fallout in followup PRs

…asources

kibanamachine · 2020-07-01T23:12:12Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: c762e2a

Build metrics

✅ unchanged

History

💚 Build #58028 succeeded 7761eb9
💔 Build #58002 failed 56f32a2
💚 Build #57700 succeeded b312459
💚 Build #57606 succeeded 2585d2d
💚 Build #57394 succeeded 8194094

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

) * Rename `datasource` saved object to `package_config` (SO type `ingest-datasource` to `ingest-package-config`) and adjust mappings: - Remove unused `processors` field on input & stream levels - Remove unnecessary `enabled` field on package config & input levels - Rename `agent_stream` field to `compiled_stream` - Reorder other fields so that important fields are closer to top & similar fields are grouped together Also, - Remove 7.9.0 migrations as we are not supporting an upgrade path from experimental to beta release * Pluralize `ingest-package-configs` for consistency * Rename `Datasource`-related types to `PackageConfig`, update all references (does not include actual type definition changes yet) * Rename `Datasource` schemas and rest spec typings to `PackageConfig` (does not include actual schema changes yet) * Change `datasources` on agent config typings and schemas to `package_configs` and update all references * Add back `enabled` field on package config and input levels. They are needed for current & future UI features. Also: - Match types and schemas with saved object mappings (`agent_stream` to `compiled_stream`, removal of `processors`) - Set `namespace` to be a required property on agent config and package config types, add validation support for it on UI * Rename server-side datasource references in file names, variable names, and routes * Update spec file and schema file * Update doc wording * Rename all instances of datasource in file paths and variable names on client sides, and for Endpoint too * Minor copy adjustments, fix i18n check * Replace datasource references in tests and fixtures; remove unused `ingest/policies` es archiver data * Fix tests * Fix test field name * Fix test fixtures fields again * Fix i18n # Conflicts: # x-pack/plugins/translations/translations/ja-JP.json # x-pack/plugins/translations/translations/zh-CN.json

* master: (46 commits) [Visualize] Add missing advanced settings and custom label for pipeline aggs (elastic#69688) Use dynamic: false for config saved object mappings (elastic#70436) [Ingest Pipelines] Error messages (elastic#70167) [APM] Show transaction rate per minute on Observability Overview page (elastic#70336) Filter out error when calculating a label (elastic#69934) [Visualizations] Each visType returns its supported triggers (elastic#70177) [Telemetry] Report data shippers (elastic#64935) Reduce SavedObjects mappings for Application Usage (elastic#70475) [Lens] fix dimension label performance issues (elastic#69978) Skip failing endgame tests (elastic#70548) [SIEM] Reenabling Cypress tests (elastic#70397) [SIEM][Security Solution][Endpoint] Endpoint Artifact Manifest Management + Artifact Download and Distribution (elastic#67707) [Security] Adds field mapping support to rule creation (elastic#70288) SECURITY-ENDPOINT: add fields for events to metadata document (elastic#70491) Fixed assertion in hybrid index pattern test to iterate through indices (elastic#70130) [SIEM][Exceptions] - Exception builder component (elastic#67013) [Ingest Manager] Rename data sources to package configs (elastic#70259) skip suites blocking es snapshot promomotion (elastic#70532) [Metrics UI] Fix asynchronicity and error handling in Snapshot API (elastic#70503) fix export response (elastic#70473) ...

jen-huang self-assigned this Jun 29, 2020

jen-huang added 12 commits June 29, 2020 14:39

Pluralize ingest-package-configs for consistency

482eebb

Rename Datasource-related types to PackageConfig, update all refe…

f000015

…rences (does not include actual type definition changes yet)

Rename Datasource schemas and rest spec typings to PackageConfig …

77cebc2

…(does not include actual schema changes yet)

Change datasources on agent config typings and schemas to `package_…

6e7320e

…configs` and update all references

Rename server-side datasource references in file names, variable name…

58e5e86

…s, and routes

Update spec file and schema file

4b31a66

Update doc wording

849f228

Rename all instances of datasource in file paths and variable names o…

7972d2b

…n client sides, and for Endpoint too

Minor copy adjustments, fix i18n check

a985661

Replace datasource references in tests and fixtures; remove unused `i…

640c63b

…ngest/policies` es archiver data

jen-huang force-pushed the ingest/rename-datasources branch from feb5be9 to 640c63b Compare June 29, 2020 21:43

jen-huang mentioned this pull request Jun 29, 2020

[Ingest Manager] Rename and restructure data source saved object #70018

Closed

9 tasks

ruflin mentioned this pull request Jun 29, 2020

[Elastic Agent] Change structure of Elastic Agent across the Stack elastic/beats#19082

Closed

15 tasks

Fix tests

91d13f2

jen-huang mentioned this pull request Jun 29, 2020

[Ingest Manager] Name changes tracking issue #70024

Closed

jen-huang changed the title ~~[WIP][Ingest Manager] Rename data sources to package configs~~ [Ingest Manager] Rename data sources to package configs Jun 29, 2020

jen-huang added release_note:skip Skip the PR/issue when compiling release notes Team:Fleet Team label for Observability Data Collection Fleet team v7.9.0 v8.0.0 labels Jun 29, 2020

jen-huang marked this pull request as ready for review June 29, 2020 23:28

jen-huang requested a review from a team as a code owner June 29, 2020 23:28

jen-huang requested a review from a team June 29, 2020 23:28

jen-huang requested a review from a team as a code owner June 29, 2020 23:28

jen-huang commented Jun 30, 2020

View reviewed changes

Fix test field name

b459636

jen-huang added 3 commits June 29, 2020 17:11

Merge remote-tracking branch 'upstream/master' into ingest/rename-dat…

18e4bbb

…asources

Fix test fixtures fields again

8194094

Merge remote-tracking branch 'upstream/master' into ingest/rename-dat…

2585d2d

…asources

Merge remote-tracking branch 'upstream/master' into ingest/rename-dat…

b312459

…asources

kevinlog approved these changes Jul 1, 2020

View reviewed changes

jfsiii approved these changes Jul 1, 2020

View reviewed changes

jen-huang added 3 commits July 1, 2020 11:19

Merge remote-tracking branch 'upstream/master' into ingest/rename-dat…

56f32a2

…asources

Fix i18n

7761eb9

Merge remote-tracking branch 'upstream/master' into ingest/rename-dat…

c762e2a

…asources

jen-huang added release_note:skip Skip the PR/issue when compiling release notes and removed release_note:skip Skip the PR/issue when compiling release notes labels Jul 1, 2020

jen-huang merged commit 4f7da59 into elastic:master Jul 1, 2020

jen-huang deleted the ingest/rename-datasources branch July 1, 2020 23:14

This was referenced Jul 2, 2020

[7.x] [Ingest Manager] Rename data sources to package configs (#70259) #70537

Merged

[Ingest Manager] New agent config's system package doesn't use namespace #68041

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Ingest Manager] Rename data sources to package configs #70259

[Ingest Manager] Rename data sources to package configs #70259

jen-huang commented Jun 29, 2020 •

edited

Loading

elasticmachine commented Jun 29, 2020

jen-huang Jun 30, 2020

jen-huang commented Jun 30, 2020

kevinlog commented Jul 1, 2020

kevinlog left a comment

jfsiii left a comment

kibanamachine commented Jul 1, 2020

[Ingest Manager] Rename data sources to package configs #70259

[Ingest Manager] Rename data sources to package configs #70259

Conversation

jen-huang commented Jun 29, 2020 • edited Loading

Summary

Testing

Screenshots

elasticmachine commented Jun 29, 2020

jen-huang Jun 30, 2020

Choose a reason for hiding this comment

jen-huang commented Jun 30, 2020

kevinlog commented Jul 1, 2020

kevinlog left a comment

Choose a reason for hiding this comment

jfsiii left a comment

Choose a reason for hiding this comment

kibanamachine commented Jul 1, 2020

💚 Build Succeeded

Build metrics

History

jen-huang commented Jun 29, 2020 •

edited

Loading