Description
Note: I also asked this question on the elasticsearch community forum as well (https://discuss.elastic.co/t/is-field-type-wildcard-supported-in-elasticsearch-spark/275983), but the forum does not have a separate elasticsearch-hadoop section anymore (seems it had before) so I try my luck here...
What kind an issue is this?
Not sure if it is a bug or a still-missing feature but one of that.
Issue description
When reading JavaPairRDD from spark backed by elasticsearch, any fields which are of type wildcard does not appear
I understand the wildcard type was quite a recent addition to elasticsearch but I struggle to find any information if the type is supported by elasticsearch-spark.
Also when filtering it never find any hits although I know there are matches (I can find them with match searches on REST API).
Does elasticsearch-spark support the wildcard field type in elasticsearch?
Steps to reproduce
Code:
This never gets any hits:
long nrHits = JavaEsSpark
.esRDD(sc, "my_index")
.filter(record -> {
return "existing_value".equals(record._2.get("a_wildcard_field")); // notice here its a wildcard field
})
.count();
But this does:
long nrHits = JavaEsSpark
.esRDD(sc, "my_index")
.filter(record -> {
return "existing_value".equals(record._2.get("a_keyword_field")); // notice here its a keyword field
})
.count();
Version Info
Hadoop/Spark: 3.1.2
ES-Hadoop : elasticsearch-spark-30_2.12:7.13.2
ES : 7.12.1