forwardCompatibilitySchema should only strip unknown fields #204395
Description
We have observed that most if not all teams use kbn-config-schema
with strict field value validation for their model version forwardCompatibilitySchema
. Instead of only stripping away fields from newer documents that an older version might not know, this also applies strict validation which causes unintended side effects like:
- mutates the values when a schema like
duration
converts'60s' -> 'PT1M'
.
This can cause failures during a ZDT rollout since older nodes applying the forwardCompatibilitySchema validation are no longer able to read documents. To make this even more confusing, the same schema is often used forcreate
but on create validations cannot cause any mutations causing a mismatch between these code paths. - fails on partial documents retrieved with the
fields
option since not all the required fields would be present (as a result forwardCompatibilitySchema is currently skipped for partial find results) ZDT down migrations: should they still apply forwardCompatibilitySchema for partial documents #199102
While this is documented, this documentation is likely not enough because kbn-config-schema
biases developers towards strict value validation. Having to maintain two separate but largely duplicated/overlapping schemas is also inefficient.
https://github.com/elastic/kibana/blob/main/packages/core/saved-objects/core-saved-objects-server/docs/model_versions.md#L298-L299
https://github.com/elastic/kibana/blob/main/packages/core/saved-objects/core-saved-objects-server/src/model_version/schemas.ts#L48-L49
Ideally we could find a way to automatically convert a kbn-config-schema
into a validation function which only keeps the known keys but does not apply any value validation/mutation.
Activity