Unable to read date_nanos

[dlubomski]

Issue description

Unable to read date_nanos field to Spark df.
Type date reading without problems but date_nanos not show in df columns.

Strack trace:

No stack trace, code execute properly but don`t have date_nanos column in results.

Version Info

Hadoop/Spark: Spark 3.1.1
ES-Hadoop : elasticsearch-spark-30_2.12-7.12.0
ES : 7.7.1

[masseyke]

I can sort of reproduce this. I create an index with a date_nanos field:

curl -X PUT "localhost:9200/my-index?pretty" -H 'Content-Type: application/json' -d'
{
  "mappings": {
    "properties": {
      "date": {
        "type": "date_nanos" 
      }
    }
  }
}
'

Then I put some data in:

curl -X PUT "localhost:9200/my-index/_bulk?refresh&pretty" -H 'Content-Type: application/json' -d'
{ "index" : { "_id" : "1" } }
{ "date": "2015-01-01" } 
{ "index" : { "_id" : "2" } }
{ "date": "2015-01-01T12:10:30.123456789Z" } 
{ "index" : { "_id" : "3" } }
{ "date": 1420070400000 }
'

And then in spark shell:

import org.apache.spark.sql.SQLContext
val sqlContext = new SQLContext(sc)
val df = sqlContext.read.format("es").load("my-index")
df.show

And I get

Caused by: org.elasticsearch.hadoop.rest.EsHadoopParsingException: org.elasticsearch.hadoop.EsHadoopIllegalStateException: Field '_' not found; typically this occurs with arrays which are not mapped as single value
  at org.elasticsearch.hadoop.serialization.ScrollReader.readHit(ScrollReader.java:514)
  at org.elasticsearch.hadoop.serialization.ScrollReader.read(ScrollReader.java:292)
  at org.elasticsearch.hadoop.serialization.ScrollReader.read(ScrollReader.java:262)
  at org.elasticsearch.hadoop.rest.RestRepository.scroll(RestRepository.java:313)
  at org.elasticsearch.hadoop.rest.ScrollQuery.hasNext(ScrollQuery.java:93)
  at org.elasticsearch.spark.rdd.AbstractEsRDDIterator.hasNext(AbstractEsRDDIterator.scala:66)
  at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458)
  at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
  at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
  at org.apache.spark.sql.execution.WholeStageCodegenExec$$anon$1.hasNext(WholeStageCodegenExec.scala:755)
  at org.apache.spark.sql.execution.SparkPlan.$anonfun$getByteArrayRdd$1(SparkPlan.scala:345)
  at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2(RDD.scala:898)
  at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2$adapted(RDD.scala:898)
  at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
  at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373)
  at org.apache.spark.rdd.RDD.iterator(RDD.scala:337)
  at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
  at org.apache.spark.scheduler.Task.run(Task.scala:131)
  at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:497)
  at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1439)
  at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:500)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  at java.lang.Thread.run(Thread.java:748)
Caused by: org.elasticsearch.hadoop.EsHadoopIllegalStateException: Field '_' not found; typically this occurs with arrays which are not mapped as single value
  at org.elasticsearch.spark.sql.RowValueReader.rowColumns(RowValueReader.scala:51)
  at org.elasticsearch.spark.sql.RowValueReader.rowColumns$(RowValueReader.scala:48)
  at org.elasticsearch.spark.sql.ScalaRowValueReader.rowColumns(ScalaEsRowValueReader.scala:32)
  at org.elasticsearch.spark.sql.ScalaRowValueReader.createMap(ScalaEsRowValueReader.scala:69)
  at org.elasticsearch.hadoop.serialization.ScrollReader.map(ScrollReader.java:1011)
  at org.elasticsearch.hadoop.serialization.ScrollReader.read(ScrollReader.java:889)
  at org.elasticsearch.hadoop.serialization.ScrollReader.readHitAsMap(ScrollReader.java:602)
  at org.elasticsearch.hadoop.serialization.ScrollReader.readHit(ScrollReader.java:426)
  ... 23 more

Doing the same thing with a text field works as expected.

[masseyke]

@dlubomski I've created a PR to add support for date_nanos at #1803. If you're able to try it out I'd appreciate the feedback.