[Transform] transform jobs on nodes < 7.4.0 mixed with nodes > 7.4 might fail

[hendrikmuhs]

Nodes in all versions of 7.2 and 7.3 might not see transform documents created in 7.4, 7.5 ...

Only after the switch to a versioned index in #45375 (7.4.0) retrieval uses a search pattern to read back documents (config, checkpoint, state). This allows to retrieve jobs even if the cluster is partially upgraded. But if nodes are on version 7.3 and 7.4 the hardcoded index name those documents might not be retrievable.

Long story short: Mixed version clusters with nodes <7.4 together with nodes >=7.4 do not work reliable.

Solution:

• disable node allocation if any node < 7.4. This will prevent placing jobs on mixed clusters until the upgrade has completed
• document as known limitation: transforms do not execute during a rolling upgrade from 7.2/7.3 to 7.x

[elasticmachine]

Pinging @elastic/ml-core (:ml/Transform)