[CCR] Update es monitoring mapping and

[martijnvg]

change qa tests to query based on leader index.

[elasticmachine]

Pinging @elastic/es-distributed

[jasontedor]

I left one comment.

As we continue to iterate on the stats endpoint (we will) we will need to consider a way to enforce that we keep the mapping up to date with the changes?

[jasontedor]

I am not sure about this being a keyword, I think probably a numeric field type is preferred here?

[martijnvg]

Good point. It is serialized as a string now ([index-name][shard-id]). I think that is not super useful and serializing just the shard id is better? (the leader index is already stored in another field)

[jasontedor]

I think I am missing something, it should only be being serialized as the numerical shard ID, without the index name. See this test:

elasticsearch/x-pack/plugin/monitoring/src/test/java/org/elasticsearch/xpack/monitoring/collector/ccr/CcrStatsMonitoringDocTests.java

Line 142 in 23f12e4

+ "\"shard_id\":" + shardId + ","

and the x-content logic:

elasticsearch/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/ccr/ShardFollowNodeTaskStatus.java

Line 380 in 23f12e4

builder.field(SHARD_ID.getPreferredName(), shardId);

[martijnvg]

whoops. I read shardId as field name and I assumed ShardId class.

[martijnvg]

As we continue to iterate on the stats endpoint (we will) we will need to consider a way to enforce that we keep the mapping up to date with the changes?

Agreed. I'm not sure how though. Add unit test in monitoring that iterates over all static parse fields, parse the mapping and then verify if there is a entry in the mapping?

[ycombinator]

Maybe its too heavy handed but Monitoring has integration tests. Maybe one of those to collect stats and then query for them, to ensure that all the fields were indeed indexed as expected?

[martijnvg]

@ycombinator Thanks, I did not know that. I think we can add a test here or somewhere else that uses CcrStatsCollector directly, get a monitoring doc based on a dummy ccr stat response, index that monitor doc via an Exporter and then expect that we can search on all field that are in ShardFollowNodeTaskStatus.

[jasontedor]

LGTM.