[Transform] Possible NPE in usage endpoint

[hendrikmuhs]

This has been reported on a 7.3 installation:

java.lang.NullPointerException: null
        at org.elasticsearch.xpack.dataframe.DataFrameFeatureSet.usage(DataFrameFeatureSet.java:120) ~[?:?]
        at org.elasticsearch.xpack.core.action.TransportXPackUsageAction.lambda$masterOperation$0(TransportXPackUsageAction.java:70) ~[?:?]
        at org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:117) ~[?:?]
        at org.elasticsearch.xpack.core.action.TransportXPackUsageAction$2.onResponse(TransportXPackUsageAction.java:75) ~[?:?]

First investigation:

The stack points to:
https://github.com/elastic/elasticsearch/blob/7.3/x-pack/plugin/data-frame/src/main/java/org/elasticsearch/xpack/dataframe/DataFrameFeatureSet.java#L120

taskState seems to be null, which is strange, because there is no place it's set to null. My best guess at the moment:

https://github.com/elastic/elasticsearch/blob/7.3/x-pack/plugin/data-frame/src/main/java/org/elasticsearch/xpack/dataframe/transforms/DataFrameTransformTask.java#L100

This set's the taskState from a previous State which is loaded from the internal index. The state document is loaded in a lenient fashion, so I can only imagine that this got somehow corrupted and the task state got set to null at the above place.

Solution

It's hard to find a definitive fix as a lot of changes happened between 7.3 and now. Its also not clear how to reproduce the problem. Nevertheless based on the assumptions made we should harden the code and make it robust against potential corruption.

[elasticmachine]

Pinging @elastic/ml-core (:ml/Transform)

[hendrikmuhs]

Another occurrence after transform has been disabled on a subset of nodes.

[2020-02-07T09:10:03,037][ERROR][o.e.x.m.c.c.ClusterStatsCollector] [XXXX] collector [cluster_stats] failed to collect data
java.lang.NullPointerException: null
	at org.elasticsearch.xpack.transform.TransformFeatureSet.usage(TransformFeatureSet.java:120) ~[?:?]
	at org.elasticsearch.xpack.core.action.TransportXPackUsageAction.lambda$masterOperation$0(TransportXPackUsageAction.java:72) ~[?:?]
	at org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:117) ~[?:?]
	at org.elasticsearch.xpack.core.action.TransportXPackUsageAction$2.onResponse(TransportXPackUsageAction.java:77) ~[?:?]
	at org.elasticsearch.xpack.core.action.TransportXPackUsageAction$2.onResponse(TransportXPackUsageAction.java:72) ~[?:?]