[ML] Jobs do not recover if the cluster state is deleted

[elasticmachine]

Original comment by @sophiec20:

There are "rare" occasions when the cluster state may be manually deleted. For example, if a new trial license is wanted, or if the cluster state is red and cannot be recovered. This is not a recommended way to recover, but might be the last ditch option to try in some circumstances.

If the cluster state is removed rm -rf _state then upon an elasticsearch restart, the indices are recreated based upon files seen on disk.

If this occurs, ML job and datafeed configurations are not recreated (amongst other things). However the results and state indices are. It would not be possible to view these ML jobs without some black magic in play, and knowledge of the exact ML configuration details.

There are no plans yet to address or mitigate this, nor any easy resolution. Noting for informational purposes.

[dimitris-athanasiou]

I think this will be solved when we move the jobs/datafeeds off the cluster state.

[davidkyle]

ML jobs are now stored in the index. Closed by #32905