Closed
Description
Original comment by @droberts195:
If we're going to introduce completely new job types in the future, we need to change the way unknown job/datafeed cluster state is validated.
While trying to add categorizer
jobs, which are quite similar to anomaly_detector
jobs, I ran into the following problem:
- Logically, a
categorizer
job should have nodetectors
- But the
AnalysisConfig
class requiresdetectors
- There are two possible solutions that seem reasonable at first glance:
- Have
categorizer
jobs have acategorization_config
instead ofanalysis_config
- Change
analysis_config
so thatdetectors
is not required if thejob_type
iscategorizer
- Have
- Unfortunately neither of these works:
- Old nodes will ignore
categorization_config
when parsing metadata, but then error becauseJob
requires ananalysis_config
- Old nodes will not tolerate an
analysis_config
with no detectors
- Old nodes will ignore
- This results in the messy solution that
categorizer
jobs will have to have ananalysis_config
that includes unnecessary fields - new nodes will ignore these fields and mask them when printing the config in REST responses, but old nodes will show the unnecessary bits
I think the only long term solution that allows the necessary degree of extensibility is to hold Job
s as arbitrary Map<String, Object>
or BytesReference
when parsing from cluster state, and only interpret what's in the Map
or BytesReference
if the job_type
is understood. This is pretty much how index settings work.