packages_idm.mdx

id	slug	title	description	tags
kibDevDocsOpsPackages	/kibana-dev-docs/ops/packages	Packages / Internal Dependency Management	Information about packages, where we are going, and how we are going to get there	kibana dev contributor operations idm packages

Summary

The size of the Kibana repository has surpassed almost all other Typescript projects on Github, and the Javascript tooling ecosystem is inadequate. AppEx Operations team has done a lot of work over the years to close these gaps and keep up with our codebase's growth. Still, significant steps are necessary to provide a more efficient development experience. The AppEx Operations team is leading an effort to migrate to Bazel, which among other things, will provide remote caching and incremental builds. This initiative should drastically improve the productivity of Kibana contributors by minimizing what needs to be built, both locally and in CI, and providing faster and more thorough feedback.

This document represents the target of the IDM project and not the currently implemented features. See [What works now?][status] for information about the current implementation status.

Goals

• Use packages as a core unit of code used throughout the repository
• Packages have well-defined boundaries, a single responsibility, and are easily reusable and shareable across the repository when desired
• Support organizing the repository by domain
• Allow developers and CI to benefit from a remote cache and incremental builds for all development/validation/build tasks

Definitions

These are some of the terms we are using to describe parts of this initiative:

Build Tasks/Tasks : We refer to any task/command that is executed as part of development or CI as a "build task" or just "task".

Package : Packages can be installed from NPM, or implemented in the repository. Local packages are structured in a specific way detailed in the Package Structure section.

Incremental tasks : The ability to execute the minimal set of build tasks necessary by inspecting the files which have changed and utilizing local+remote caches.

Package Structure

Every package has:

• a globally unique module ID in the @kbn/ namespace
• a type, see Package Types
• dependencies on other packages in the repo or from NPM declared in their kibana.jsonc file and updated automatically
• configuration files like BUILD.bazel, jest.config.js, package.json, etc. which are automatically generated when necessary from the kibana.jsonc file.
• a single interface. When you import a package you get its one (and only) interface. Packages don't have sub-imports or sub-modules.
• the ability to include jest tests alongside the code

Package Types

Package types allow us to have many different packages, pre-defined build tasks, restrictions, and unique configurations for each package type. Every package is one of the following types:

shared-common : These packages can be imported from all other packages.

shared-browser : These packages can be imported from shared-browser and public directories within plugin packages. shared-browser packages may include Storybooks.

shared-server : These packages can be imported from shared-server and server directories within plugin packages.

shared-scss : These packages can be imported by shared-browser and public directories within plugin packages, and expose an index.scss file to consumers instead of an index.ts file.

plugin : These packages were automatically created from the existing plugins at the time we switched everything over to packages. Module IDs must end with -plugin. Consumers must use import type statements.

functional-test : These packages expose one or more functional testing configurations, including API integration tests, and can not be imported by other packages. Separating functional and integration tests allows us to iterate on tests without rebuilding the application. Similarly, iterating and updating the application should mostly mean the tests don't need to rebuild.

There is currently a proposal to add a new test-helpers package type, and we expect more package types to be defined in the future if and when the need arises.

Phases

We're planning to implement the full package system in phases. Currently, those phases look like this:

Phase 1: Ground preparation

status: ✅ complete

This phase is about identifying issues that would prevent a migration for teams to provide adequate time for them before they need to migrate.

• Migrate all plugins to use module IDs
• Find instances where ESLint is disabled and illegal cross-boundary imports are used; create issues for teams to address
• Prevent naked eslint-disable
• Prevent disabling specific ESLint rules
• Windows development is not supported
• Rewrite @kbn/pm using native Node.js to remove its build step. Anything outside of bootstrap, like clean and reset commands, will move to different packages.
• Document how to break up packages into smaller units of code

Phase 2: Legacy packages migration

status: in progress

This phase is about migrating the existing legacy packages to one of the package types.

• Add kibana.json files to existing packages
• Auto-generate configuration files based on kibana.jsonc manifests
• Automate package dependency discovery

Phase 3: Double down on DX

This phase is all about making it easier for teams to start breaking their plugin up into packages.

• Rebuilding packages when running the Kibana Development CLI is automatic
• Pause requests when packages are being re-built
• Ensure the server restarts when files in a shared-server package are rebuilt
• Package linting to validate rules
• Start with a package rule to validate dependencies are all used and included
• Document how to create, build and watch packages.

Phase 4: Plugins everywhere

This phase is all about supporting the creation of plugin-browser and plugin-server packages anywhere in the repository.

• Extracting Webpack config from @kbn/optimizer to work on a single bundle inside Bazel
• Update plugin discovery to find and differentiate between legacy plugins and plugin-browser/server packages
• ESLint rules to validate imports into and out of plugin-browser and plugin-server packages
• Finish the migration of core to packages and reflect with the core team (and possibly the Shared UX team) on what we should recommend for plugin authors before we start to migrate legacy plugins to the package system
• Documentation, documentation, documentation
• How do 3rd party plugins migrate to the new system?

Phase 5: Legacy plugins migration

This phase is all about having the solution teams migrate their legacy plugins into packages.

• Identify the order of plugins that can be migrated
• Identify and communicate what needs to be done by teams
• Provide migration consultations
• Deprecate legacy 3rd party plugin styles and communicate 18 month migration period

Phase 6: Cleanup

This phase is about finalizing the rough edges and making sure every piece of code is on Bazel.

• No code lives outside a Bazel package
• Extend the package development tooling to support 3rd party package development and allow packages to participate in the benefits of Bazel within the plugins directory.
• Ability for 3rd party packages to require specific versions of specific packages from NPM
• Automatically build the components that need changes
• Build package artifacts that can be installed in a Kibana distributable

FAQ

Is it time for me to start creating packages?

Probably not. The Shared UX and Core teams are currently our "Guinea Pig" teams and they're experiencing the pain of living on the bleeding edge. If you want to create a single package you are welcome to, but for now, it's probably best that you wait until Operations reached out to your team.

How do circular dependencies work?

By breaking the repository into packages we can't support cross-package circular dependencies.

For example, imagine trying to build the types for @kbn/a that depend on the types for @kbn/b. If @kbn/b also depends on the types for @kbn/a there's a circular dependency meaning there isn't a way to build the types for either.

If you cause a circular dependency in the task graph, Bazel will produce a pretty great error message explaining where the cycle is.

The solution to resolving a circular dependency has, thus far, been to break out the component causing it, into a separate package. Packages are lightweight and are very easy to create (work in progress) so please feel comfortable creating more packages as you need to.

How do we name packages?

There are a few package naming rules:

• all packages must use the @kbn/ namespace
• plugin-type packages must end with -plugin
• considering that we operate in a global namespace, avoid overly generic names

Other than these rules, it's up to you and your team to decide on an appropriate name for your package.

Keep the single responsibility principle in mind. A good indication that a package contains too much is that naming it isn’t easy. In such cases, consider splitting it into multiple, smaller packages with their specific purpose.

The shared-ux team makes a lot of packages, each containing a single, widely shared component along with helpers and types, for other packages to consume. The shared-ux team has used the following naming scheme:

/{domain}/{componentName}
  impl/  :: `@kbn/shared-ux-{domain}-{componentName}`
  mocks/ :: `@kbn/shared-ux-{domain}-{componentName}-mocks`
  types/ :: `@kbn/shared-ux-{domain}-{componentName}-types`

The @kbn/{team}-{domain}-{component}(-{type})? style naming scheme is also followed by the core team in their packages:

• @kbn/core-analytics-browser
• @kbn/core-analytics-browser-internal
• @kbn/core-analytics-browser-mocks
• @kbn/core-analytics-server
• @kbn/core-analytics-server-mocks
• @kbn/core-analytics-server-internal
• etc.

Where do I put my package?

The only rule the package system enforces is that packages can't live inside of other packages. Additionally, for licensing purposes, it's probably best to keep SSPL licensed code in the packages or src directories and Elastic licensed code in the x-pack directory.

Otherwise, you can put your packages wherever you like. It's probably best that you don't put them in the current packages directory as it's huge and only getting bigger with time.

To define a new directory where packages will live you will need to edit the BAZEL_PACKAGE_DIRS const. This list points to all the directories where packages can live in the repository and includes the current list of locations where packages are being created.

What works now?

Today we have a simplistic package generator that produces legacy package definitions. We've finished laying the groundwork and have both the Core and Shared UX teams experimenting with it. You can use the legacy package generator with node scripts/generate package and create a package, however, we don’t recommend other teams to start migrating large portions of their code to packages just yet.

We're now entering Phase 2 of the plan, more details about the phases of our plan can be found above