Support for all Elasticsearch field types

[masseyke]

There are quite a few field types that have been added to Elasticsearch that are not currently supported in es-hadoop.

Unsupported field types

• aggregate_metric_double
• alias
• annotated_text
• completion (ignored)
• constant_keyword
• date_nanos Unable to read date_nanos #1653
• flattened (does not cause a failure on read, but they do not come back)
• histogram
• ip Elasticsearch-Spark read data contains fields of IP type, triggering type cast error. #1737
• match_only_text
• murmur3
• percolator
• point
• range
• rank_feature
• rank_features
• search_as_you_type
• shape
• version
• wildcard Is elasticsearch field type "wildcard" supported in elasticsearch-spark? #1697
• dynamic_date_formats Reading into spark dataframe from indexes with dynamic field mapping configuration #1988
• dense_vector

I have not tested all of them, but at least some of these cause failures in es-hadoop if you read from an index that uses them.

[rseldner]

Looks like the dense_vector type needs to be added as well.
https://www.elastic.co/guide/en/elasticsearch/reference/current/dense-vector.html

[sbhatm1213]

Has the dense_vector field type been added ? Or does anyone know of any other way to pull that field on Pyspark ?

[mouhagaye]

I've tested recently with spark reading from index containing most of elastic search field type.
Used versions:

• Spark : 3.3.1
• ElasticSearch: 8.12.1
• elasticsearch-spark: 8.12.1

Spark read successfully the index but the these following field type does not appear on the created dataframe:

• constant_keyword
• unsigned_long
• flattened
• integer_range
• float_range
• long_range
• double_range
• date_range
• ip_range
• version
• agg_metric
• histogram
• match_only_text
• completion
• search_as_you_type
• dense_vector
• sparse_vector
• rank_feature
• rank_features
• point
• shape
• percolator

[berglh]

I hit this issue with match_only_text field using Spark 3.4.1 and the elasticsearch-spark-30_2.12-8.7.1.jar against Elasticsearch 8.11 - ouch. Just want to return the message field, which is using the default dynamic template which seems to default to this type.

Edit: I also tried to append keyword field to message via the index template and rolled over the data stream, and Elasticsearch Spark seems to still ignore the keyword value.