Unable to persist into HBase using a custom coder

[Srinathc]

Tried to persist a field of type "java.sql.Timestamp" by using a custom coder.
The instantiation of the coder fails with error "IllegalArgumentException: argument type mismatch".

Attaching sample code snippet.
SessionWriter.scala.txt

[Srinathc]

Narrowed down the issue to: core/src/main/scala/org/apache/spark/sql/execution/datasources/hbase/types/SHCDataType.scala

The methods getConstructor and newInstance accept variable arguments.
While invoking from scala it should be converted to varargs reference.
All unit tests passed with the fix below. Will submit a pull request.

@@ -72,8 +72,8 @@ object SHCDataTypeFactory {
     } else {
       // Data type implemented by user
       Class.forName(f.fCoder)
-        .getConstructor(classOf[Option[Field]])
-        .newInstance(Some(f))
+        .getConstructor(Array(classOf[Option[Field]]):_*)
+        .newInstance(Array(Some(f)):_*)
         .asInstanceOf[SHCDataType]
     }
   }

[Srinathc]

Submitted a pull request #138

[weiqingy]

Thanks, @Srinathc.
Which shc version you were using? Based on your catalog definition, you did not specify "tableCoder" and "version" (refer: here), then your case would use "PrimitiveType" as "tableCoder" (tableCoder is used to encode/decode every column in the table except for the columns you specify the filed coder for them, e.g, start and end in your catalog). However, currently SHC only support Avro as filed coder (refer here and here). I am wondering how your application worked? Were you using SHC v1.0.0 or v1.0.1 which support coder for each column refer here ?

[Srinathc]

Hi @weiqingy ,
I was using the version 1.1.0-2.1-s_2.11.
I wasn't aware that only Avro was supported and I observed from the code that there is a validation check to ensure that at least one coder is Avro coder. So I had locally commented it out to get it going. I wasn't sure why there was a restriction and thought of checking it out later.

[weiqingy]

I see. Yes, for your case, I think you can comment it out to make it work. The reason why shc puts this restriction here is if the columns in the same hbase table have two different coders, e.g. one is PrimitiveType and one is Phoenix type, whose byte orders in Hbase are very different, that may bring issues to users. I am considering removing this restriction.

@Srinathc Thanks for submitting PR. I'll review it very soon.