build immutable bundles for new platform plugins (#53976)

* build immutable bundles for new platform plugins

* only inspect workers if configured to do so

* [navigation] use an index.scss file

* add yarn.lock symlink

* set pluginScanDirs in test so fixtures stay consistent

* cleanup helpers a little

* fix type error

* support KBN_OPTIMIZER_MAX_WORKERS for limiting workers via env

* test support for KBN_OPTIMIZER_MAX_WORKERS

* expand the available memory for workers when only running one or two

* add docs about KBN_OPTIMIZER_MAX_WORKERS environment variable

* fix README link

* update kbn/pm dist

* implement bundle caching/reuse

* update kbn/pm dist

* don't check for cache if --no-cache is passed

* update renovate config

* standardize on index.scss, move console styles over

* add support for --no-cache to cli

* include worker config vars in optimizer version

* ignore concatenated modules

* update integration test

* add safari to browserslist to avoid user-agent warnings in dev

* update docs, clean up optimizer message/misc naming

* always handle initialized messages, don't ignore states that are attached to specific events

* reword caching docs, add environment var to disable caching

* tweak logging and don't use optimizer.useBundleCache as that's disabled in dev

* handle change notifications

* batch changes for 1 second

* rename CompilerState type to CompilerMsg

* getChanges() no longer needs to assign changes to dirs

* remove unused deps

* split up run_worker.ts and share cacheKey generation logic

* add a couple docs

* update tests and remove unused imports

* specify files when creating bundle cache key

* remove one more unused import

* match existing dev cli output more closely

* update kbn/pm dist

* set KBN_NP_PLUGINS_BUILT to avoid warning in CI

* avoid extending global window type

* add note to keep pluginScanDirs in sync

* pass browserslistEnv in workerConfig so it is used for cache key

* load commons.bundle.js in parallel too

* emit initialized+success states if all bundles are cached

* load bootstraps as quickly as possible

* skip flaky suite

* bump

* update jest snapshots

* remove hashing from cache key generation

* remove unnecessary non-null assertion

* improve docs and break up Optimizer#run()

* remove unused import

* refactor kbn/optimizer to break up observable logic, implement more helpful cache invalidation logic with logging

* fix tests

* add initializing phase

* avoid rxjs observable constructor

* remove unnecessary rxjs helper, add tests for bundle cache

* update consumers of optimizer

* update readme with new call style

* replace "new platform" with "kibana platform"

* fix a couple more renames

* add support for several plain-text file formats

* fix naming of OptimizerMsg => OptimizerUpdate, use "store" naming too

* one more OptimizerMsg update

* ensure bundles are not cached when cache config is false

* test for initializing states and bundle cache events

* remove unnecessary timeout change

* Remove unnecessary helpers

* Add tests for BundleCache class

* Add tests for Bundle class

* test summarizeEvent$

* missing paths are no longer listed in mtimes map

* add tests for optimizer/cache_keys

* Add some extra docs

* Remove labeled loop

* add integration test for kbn-optimizer watcher components

* querystring-browser removed

* tweak logging a smidge, improve info and final message

* remove unused imports

* remove duplication of getModuleCount() method

* move type annotation that validates things

* clear up the build completion message

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

Author: Spencer

.browserslistrc

@@ -1,3 +1,9 @@
+[production]
 last 2 versions
 > 5%
 Safari 7 # for PhantomJS support: https://github.com/elastic/kibana/issues/27136
+
+[dev]
+last 1 chrome versions
+last 1 firefox versions
+last 1 safari versions

.github/CODEOWNERS

@@ -103,6 +103,7 @@
 /packages/*babel*/ @elastic/kibana-operations
 /packages/kbn-dev-utils*/ @elastic/kibana-operations
 /packages/kbn-es/ @elastic/kibana-operations
+/packages/kbn-optimizer/ @elastic/kibana-operations
 /packages/kbn-pm/ @elastic/kibana-operations
 /packages/kbn-test/ @elastic/kibana-operations
 /packages/kbn-ui-shared-deps/ @elastic/kibana-operations

package.json

@@ -137,12 +137,6 @@
     "@kbn/test-subj-selector": "0.2.1",
     "@kbn/ui-framework": "1.0.0",
     "@kbn/ui-shared-deps": "1.0.0",
-    "@types/flot": "^0.0.31",
-    "@types/json-stable-stringify": "^1.0.32",
-    "@types/lodash.clonedeep": "^4.5.4",
-    "@types/node-forge": "^0.9.0",
-    "@types/react-grid-layout": "^0.16.7",
-    "@types/recompose": "^0.30.5",
     "JSONStream": "1.3.5",
     "abort-controller": "^3.0.0",
     "angular": "^1.7.9",
@@ -152,11 +146,12 @@
     "angular-route": "^1.7.9",
     "angular-sanitize": "^1.7.9",
     "angular-sortable-view": "^0.0.17",
-    "autoprefixer": "9.6.1",
+    "autoprefixer": "^9.7.4",
     "babel-loader": "^8.0.6",
     "bluebird": "3.5.5",
     "boom": "^7.2.0",
     "brace": "0.11.1",
+    "browserslist-useragent": "^3.0.2",
     "cache-loader": "^4.1.0",
     "chalk": "^2.4.2",
     "check-disk-space": "^2.1.0",
@@ -165,7 +160,7 @@
     "commander": "3.0.2",
     "compare-versions": "3.5.1",
     "core-js": "^3.2.1",
-    "css-loader": "2.1.1",
+    "css-loader": "^3.4.2",
     "d3": "3.5.17",
     "d3-cloud": "1.2.5",
     "deep-freeze-strict": "^1.1.1",
@@ -226,7 +221,7 @@
     "opn": "^5.5.0",
     "oppsy": "^2.0.0",
     "pegjs": "0.10.0",
-    "postcss-loader": "3.0.0",
+    "postcss-loader": "^3.0.0",
     "prop-types": "15.6.0",
     "proxy-from-env": "1.0.0",
     "pug": "^2.0.4",
@@ -259,7 +254,7 @@
     "seedrandom": "^3.0.5",
     "semver": "^5.5.0",
     "style-it": "^2.1.3",
-    "style-loader": "0.23.1",
+    "style-loader": "^1.1.3",
     "symbol-observable": "^1.2.0",
     "tar": "4.4.13",
     "terser-webpack-plugin": "^2.3.4",
@@ -279,7 +274,7 @@
     "vega-schema-url-parser": "1.0.0",
     "vega-tooltip": "^0.12.0",
     "vision": "^5.3.3",
-    "webpack": "4.41.0",
+    "webpack": "^4.41.5",
     "webpack-merge": "4.2.2",
     "whatwg-fetch": "^3.0.0",
     "wrapper-webpack-plugin": "^2.1.0",
@@ -300,6 +295,7 @@
     "@kbn/eslint-plugin-eslint": "1.0.0",
     "@kbn/expect": "1.0.0",
     "@kbn/plugin-generator": "1.0.0",
+    "@kbn/optimizer": "1.0.0",
     "@kbn/test": "1.0.0",
     "@kbn/utility-types": "1.0.0",
     "@microsoft/api-documenter": "7.7.2",
@@ -312,6 +308,7 @@
     "@types/babel__core": "^7.1.2",
     "@types/bluebird": "^3.1.1",
     "@types/boom": "^7.2.0",
+    "@types/browserslist-useragent": "^3.0.0",
     "@types/chance": "^1.0.0",
     "@types/cheerio": "^0.22.10",
     "@types/chromedriver": "^2.38.0",
@@ -324,6 +321,7 @@
     "@types/enzyme": "^3.9.0",
     "@types/eslint": "^6.1.3",
     "@types/fetch-mock": "^7.3.1",
+    "@types/flot": "^0.0.31",
     "@types/getopts": "^2.0.1",
     "@types/glob": "^7.1.1",
     "@types/globby": "^8.0.0",
@@ -337,17 +335,20 @@
     "@types/joi": "^13.4.2",
     "@types/jquery": "^3.3.31",
     "@types/js-yaml": "^3.11.1",
+    "@types/json-stable-stringify": "^1.0.32",
     "@types/json5": "^0.0.30",
     "@types/license-checker": "15.0.0",
     "@types/listr": "^0.14.0",
     "@types/lodash": "^3.10.1",
+    "@types/lodash.clonedeep": "^4.5.4",
     "@types/lru-cache": "^5.1.0",
     "@types/markdown-it": "^0.0.7",
     "@types/minimatch": "^2.0.29",
     "@types/mocha": "^5.2.7",
     "@types/moment-timezone": "^0.5.12",
     "@types/mustache": "^0.8.31",
     "@types/node": "^10.12.27",
+    "@types/node-forge": "^0.9.0",
     "@types/numeral": "^0.0.26",
     "@types/opn": "^5.1.0",
     "@types/pegjs": "^0.10.1",
@@ -357,11 +358,13 @@
     "@types/reach__router": "^1.2.6",
     "@types/react": "^16.9.11",
     "@types/react-dom": "^16.9.4",
+    "@types/react-grid-layout": "^0.16.7",
     "@types/react-redux": "^6.0.6",
     "@types/react-resize-detector": "^4.0.1",
     "@types/react-router": "^5.1.3",
     "@types/react-router-dom": "^5.1.3",
     "@types/react-virtualized": "^9.18.7",
+    "@types/recompose": "^0.30.6",
     "@types/redux": "^3.6.31",
     "@types/redux-actions": "^2.6.1",
     "@types/request": "^2.48.2",
@@ -461,7 +464,7 @@
     "pixelmatch": "^5.1.0",
     "pkg-up": "^2.0.0",
     "pngjs": "^3.4.0",
-    "postcss": "^7.0.5",
+    "postcss": "^7.0.26",
     "postcss-url": "^8.0.0",
     "prettier": "^1.19.1",
     "proxyquire": "1.8.0",

packages/kbn-dev-utils/src/index.ts

@@ -18,12 +18,7 @@
  */
 
 export { withProcRunner, ProcRunner } from './proc_runner';
-export {
-  ToolingLog,
-  ToolingLogTextWriter,
-  pickLevelFromFlags,
-  ToolingLogCollectingWriter,
-} from './tooling_log';
+export * from './tooling_log';
 export { createAbsolutePathSerializer } from './serializers';
 export {
   CA_CERT_PATH,

packages/kbn-dev-utils/src/serializers/absolute_path_serializer.ts

@@ -17,7 +17,9 @@
  * under the License.
  */
 
-export function createAbsolutePathSerializer(rootPath: string) {
+import { REPO_ROOT } from '../repo_root';
+
+export function createAbsolutePathSerializer(rootPath: string = REPO_ROOT) {
   return {
     print: (value: string) => value.replace(rootPath, '<absolute path>').replace(/\\/g, '/'),
     test: (value: any) => typeof value === 'string' && value.startsWith(rootPath),

packages/kbn-dev-utils/src/tooling_log/index.ts

@@ -19,5 +19,5 @@
 
 export { ToolingLog } from './tooling_log';
 export { ToolingLogTextWriter, ToolingLogTextWriterConfig } from './tooling_log_text_writer';
-export { pickLevelFromFlags, LogLevel } from './log_levels';
+export { pickLevelFromFlags, parseLogLevel, LogLevel } from './log_levels';
 export { ToolingLogCollectingWriter } from './tooling_log_collecting_writer';

packages/kbn-dev-utils/src/tooling_log/tooling_log_text_writer.ts

@@ -82,20 +82,28 @@ export class ToolingLogTextWriter implements Writer {
     }
   }
 
-  write({ type, indent, args }: Message) {
-    if (!shouldWriteType(this.level, type)) {
+  write(msg: Message) {
+    if (!shouldWriteType(this.level, msg.type)) {
       return false;
     }
 
-    const txt = type === 'error' ? stringifyError(args[0]) : format(args[0], ...args.slice(1));
-    const prefix = has(MSG_PREFIXES, type) ? MSG_PREFIXES[type] : '';
+    const prefix = has(MSG_PREFIXES, msg.type) ? MSG_PREFIXES[msg.type] : '';
+    ToolingLogTextWriter.write(this.writeTo, prefix, msg);
+    return true;
+  }
+
+  static write(writeTo: ToolingLogTextWriter['writeTo'], prefix: string, msg: Message) {
+    const txt =
+      msg.type === 'error'
+        ? stringifyError(msg.args[0])
+        : format(msg.args[0], ...msg.args.slice(1));
 
     (prefix + txt).split('\n').forEach((line, i) => {
       let lineIndent = '';
 
-      if (indent > 0) {
+      if (msg.indent > 0) {
         // if we are indenting write some spaces followed by a symbol
-        lineIndent += ' '.repeat(indent - 1);
+        lineIndent += ' '.repeat(msg.indent - 1);
         lineIndent += line.startsWith('-') ? '└' : '│';
       }
 
@@ -105,9 +113,7 @@ export class ToolingLogTextWriter implements Writer {
         lineIndent += PREFIX_INDENT;
       }
 
-      this.writeTo.write(`${lineIndent}${line}\n`);
+      writeTo.write(`${lineIndent}${line}\n`);
     });
-
-    return true;
   }
 }

packages/kbn-eslint-import-resolver-kibana/package.json

@@ -16,6 +16,6 @@
     "glob-all": "^3.1.0",
     "lru-cache": "^4.1.5",
     "resolve": "^1.7.1",
-    "webpack": "^4.41.0"
+    "webpack": "^4.41.5"
   }
 }

packages/kbn-interpreter/package.json

@@ -23,15 +23,15 @@
     "@kbn/dev-utils": "1.0.0",
     "babel-loader": "^8.0.6",
     "copy-webpack-plugin": "^5.0.4",
-    "css-loader": "2.1.1",
+    "css-loader": "^3.4.2",
     "del": "^5.1.0",
     "getopts": "^2.2.4",
     "pegjs": "0.10.0",
-    "sass-loader": "^7.3.1",
-    "style-loader": "0.23.1",
+    "sass-loader": "^8.0.2",
+    "style-loader": "^1.1.3",
     "supports-color": "^7.0.0",
     "url-loader": "2.2.0",
-    "webpack": "4.41.0",
-    "webpack-cli": "^3.3.9"
+    "webpack": "^4.41.5",
+    "webpack-cli": "^3.3.10"
   }
 }

packages/kbn-optimizer/README.md

@@ -0,0 +1,110 @@
+# @kbn/optimizer
+
+`@kbn/optimizer` is a package for building Kibana platform UI plugins (and hopefully more soon).
+
+Kibana Platform plugins with `"ui": true` in their `kibana.json` file will have their `public/index.ts` file (and all of its dependencies) bundled into the `target/public` directory of the plugin. The build output does not need to be updated when other plugins are updated and is included in the distributable without requiring that we ship `@kbn/optimizer` 🎉.
+
+## Webpack config
+
+The [Webpack config][WebpackConfig] is designed to provide the majority of what was available in the legacy optimizer and is the same for all plugins to promote consistency and keep things sane for the operations team. It has support for JS/TS built with babel, url imports of image and font files, and support for importing `scss` and `css` files. SCSS is pre-processed by [postcss][PostCss], built for both light and dark mode and injected automatically into the page when the parent module is loaded (page reloads are still required for switching between light/dark mode). CSS is injected into the DOM as it is written on disk when the parent module is loaded (no postcss support).
+
+Source maps are enabled except when building the distributable. They show the code actually being executed by the browser to strike a balance between debuggability and performance. They are not configurable at this time but will be configurable once we have a developer configuration solution that doesn't rely on the server (see [#55656](https://github.com/elastic/kibana/issues/55656)).
+
+### IE Support
+
+To make front-end code easier to debug the optimizer uses the `BROWSERSLIST_ENV=dev` environment variable (by default) to build JS and CSS that is compatible with modern browsers. In order to support older browsers like IE in development you will need to specify the `BROWSERSLIST_ENV=production` environment variable or build a distributable for testing.
+
+## Running the optimizer
+
+The `@kbn/optimizer` is automatically executed from the dev cli, the Kibana build scripts, and in CI. If you're running Kibana locally in some other way you might need to build the plugins manually, which you can do by running `node scripts/build_kibana_platform_plugins` (pass `--help` for options).
+
+### Worker count
+
+You can limit the number of workers the optimizer uses by setting the `KBN_OPTIMIZER_MAX_WORKERS` environment variable. You might want to do this if your system struggles to keep up while the optimizer is getting started and building all plugins as fast as possible. Setting `KBN_OPTIMIZER_MAX_WORKERS=1` will cause the optimizer to take the longest amount of time but will have the smallest impact on other components of your system.
+
+We only limit the number of workers we will start at any given time. If we start more workers later we will limit the number of workers we start at that time by the maximum, but we don't take into account the number of workers already started because it is assumed that those workers are doing very little work. This greatly simplifies the logic as we don't ever have to reallocate workers and provides the best performance in most cases.
+
+### Caching
+
+Bundles built by the the optimizer include a cache file which describes the information needed to determine if the bundle needs to be rebuilt when the optimizer is restarted. Caching is enabled by default and is very aggressive about invalidating the cache output, but if you need to disable caching you can pass `--no-cache` to `node scripts/build_kibana_platform_plugins`, or set the `KBN_OPTIMIZER_NO_CACHE` environment variable to anything (env overrides everything).
+
+When a bundle is determined to be up-to-date a worker is not started for the bundle. If running the optimizer with the `--dev/--watch` flag, then all the files referenced by cached bundles are watched for changes. Once a change is detected in any of the files referenced by the built bundle a worker is started. If a file is changed that is referenced by several bundles then workers will be started for each bundle, combining workers together to respect the worker limit.
+
+## API
+
+To run the optimizer from code, you can import the [`OptimizerConfig`][OptimizerConfig] class and [`runOptimizer`][Optimizer] function. Create an [`OptimizerConfig`][OptimizerConfig] instance by calling it's static `create()` method with some options, then pass it to the [`runOptimizer`][Optimizer] function. `runOptimizer()` returns an observable of update objects, which are summaries of the optimizer state plus an optional `event` property which describes the internal events occuring and may be of use. You can use the [`logOptimizerState()`][LogOptimizerState] helper to write the relevant bits of state to a tooling log or checkout it's implementation to see how the internal events like [`WorkerStdio`][ObserveWorker] and [`WorkerStarted`][ObserveWorker] are used.
+
+Example:
+```ts
+import { runOptimizer, OptimizerConfig, logOptimizerState } from '@kbn/optimizer';
+import { REPO_ROOT, ToolingLog } from '@kbn/dev-utils';
+
+const log = new ToolingLog({
+  level: 'verbose',
+  writeTo: process.stdout,
+})
+
+const config = OptimizerConfig.create({
+  repoRoot: Path.resolve(__dirname, '../../..'),
+  watch: false,
+  oss: true,
+  dist: true
+});
+
+await runOptimizer(config)
+  .pipe(logOptimizerState(log, config))
+  .toPromise();
+```
+
+This is essentially what we're doing in [`script/build_kibana_platform_plugins`][Cli] and the new [build system task][BuildTask].
+
+## Internals
+
+The optimizer runs webpack instances in worker processes. Each worker is configured via a [`WorkerConfig`][WorkerConfig] object and an array of [`Bundle`][Bundle] objects which are JSON serialized and passed to the worker as it's arguments.
+
+Plugins/bundles are assigned to workers based on the number of modules historically seen in each bundle in an effort to evenly distribute the load across the worker pool (see [`assignBundlesToWorkers`][AssignBundlesToWorkers]).
+
+The number of workers that will be started at any time is automatically chosen by dividing the number of cores available by 3 (minimum of 2).
+
+The [`WorkerConfig`][WorkerConfig] includes the location of the repo (it might be one of many builds, or the main repo), wether we are running in watch mode, wether we are building a distributable, and other global config items.
+
+The [`Bundle`][Bundle] objects which include the details necessary to create a webpack config for a specific plugin's bundle (created using [`webpack.config.ts`][WebpackConfig]).
+
+Each worker communicates state back to the main process by sending [`WorkerMsg`][WorkerMsg] and [`CompilerMsg`][CompilerMsg] objects using IPC.
+
+The Optimizer captures all of these messages and produces a stream of update objects.
+
+Optimizer phases:
+<dl>
+  <dt><code>'initializing'</code></dt>
+  <dd>Initial phase, during this state the optimizer is validating caches and determining which builds should be built initially.</dd>
+  <dt><code>'initialized'</code></dt>
+  <dd>Emitted by the optimizer once it's don't initializing its internal state and determined which bundles are going to be built initially.</dd>
+  <dt><code>'running'</code></dt>
+  <dd>Emitted when any worker is in a running state. To determine which compilers are running, look for <code>BundleState</code> objects with type <code>'running'</code>.</dd>
+  <dt><code>'issue'</code></dt>
+  <dd>Emitted when all workers are done running and any compiler completed with a <code>'compiler issue'</code> status. Compiler issues include things like "unable to resolve module" or syntax errors in the source modules and can be fixed by users when running in watch mode.</dd>
+  <dt><code>'success'</code></dt>
+  <dd>Emitted when all workers are done running and all compilers completed with <code>'compiler success'</code>.</dd>
+  <dt><code>'reallocating'</code></dt>
+  <dd>Emitted when the files referenced by a cached bundle have changed, before the worker has been started up to update that bundle.</dd>
+</dl>
+
+Workers have several error message they may emit which indicate unrecoverable errors. When any of those messages are received the stream will error and the workers will be torn down.
+
+For an example of how to handle these states checkout the [`logOptimizerState()`][LogOptimizerState] helper.
+
+[PostCss]: https://postcss.org/
+[Cli]: src/cli.ts
+[Optimizer]: src/optimizer.ts
+[ObserveWorker]: src/observe_worker.ts
+[CompilerMsg]: src/common/compiler_messages.ts
+[WorkerMsg]: src/common/worker_messages.ts
+[Bundle]: src/common/bundle.ts
+[WebpackConfig]: src/worker/webpack.config.ts
+[BundleDefinition]: src/common/bundle_definition.ts
+[WorkerConfig]: src/common/worker_config.ts
+[OptimizerConfig]: src/optimizer_config.ts
+[LogOptimizerState]: src/log_optimizer_state.ts
+[AssignBundlesToWorkers]: src/assign_bundles_to_workers.ts
+[BuildTask]: ../../src/dev/build/tasks/build_kibana_platform_plugins.js