Skip to content

forwardCompatibilitySchema should only strip unknown fields #204395

Open
@rudolf

Description

We have observed that most if not all teams use kbn-config-schema with strict field value validation for their model version forwardCompatibilitySchema. Instead of only stripping away fields from newer documents that an older version might not know, this also applies strict validation which causes unintended side effects like:

  • mutates the values when a schema like duration converts '60s' -> 'PT1M'.
    This can cause failures during a ZDT rollout since older nodes applying the forwardCompatibilitySchema validation are no longer able to read documents. To make this even more confusing, the same schema is often used for create but on create validations cannot cause any mutations causing a mismatch between these code paths.
  • fails on partial documents retrieved with the fields option since not all the required fields would be present (as a result forwardCompatibilitySchema is currently skipped for partial find results) ZDT down migrations: should they still apply forwardCompatibilitySchema for partial documents #199102

While this is documented, this documentation is likely not enough because kbn-config-schema biases developers towards strict value validation. Having to maintain two separate but largely duplicated/overlapping schemas is also inefficient.
https://github.com/elastic/kibana/blob/main/packages/core/saved-objects/core-saved-objects-server/docs/model_versions.md#L298-L299
https://github.com/elastic/kibana/blob/main/packages/core/saved-objects/core-saved-objects-server/src/model_version/schemas.ts#L48-L49

Ideally we could find a way to automatically convert a kbn-config-schema into a validation function which only keeps the known keys but does not apply any value validation/mutation.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Assignees

No one assigned

    Labels

    Team:CoreCore services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc

    Type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions