[SPARK-12010][SQL] Add columnMapping support

[CK50]

In the past Spark JDBC write only worked with technologies which support the following INSERT statement syntax (JdbcUtils.scala: insertStatement()):

INSERT INTO $table VALUES ( ?, ?, ..., ? )

But some technologies require a list of column names:

INSERT INTO $table ( $colNameList ) VALUES ( ?, ?, ..., ? )

This was blocking the use of e.g. the Progress JDBC Driver for Cassandra.

Another limitation is that syntax 1 relies no the dataframe field ordering match that of the target table. This works fine, as long as the target table has been created by writer.jdbc().

If the target table contains more columns (not created by writer.jdbc()), then the insert fails due mismatch of number of columns or their data types.

This PR adds an extra columnMapping parameter to write.jdbc(). This optional parameter allows the user to specify how dataframe field names are mapping to target table column names.

[AmplabJenkins]

Can one of the admins verify this patch?

[srowen]

Then indent looks too deep here

[CK50]

Will do updates. - Can you please remind me of best git commands for
updating the PR without creating extra commits?

On 15.12.2015 18:59, Sean Owen wrote:

In python/pyspark/sql/readwriter.py
#10312 (comment):

     """
     if properties is None:
         properties = dict()
     jprop = JavaClass("java.util.Properties", self._sqlContext._sc._gateway._gateway_client)()
     for k in properties:
         jprop.setProperty(k, properties[k])

•    self._jwrite.mode(mode).jdbc(url, table, jprop)
  

•    if columnMapping is None:
  

•         columnMapping = dict()
  

Then indent looks too deep here

—
Reply to this email directly or view it on GitHub
https://github.com/apache/spark/pull/10312/files#r47672003.

[srowen]

Extra commits are good and fine as it helps show the evolution of the PR. They are squashed in the end. You can squash your own new several commits into one before pushing if it would more logically group and present your subsequent change.

[srowen]

There's still an extra space before the first arg in both mkString calls.
Also field => "?" could be _ => "?" for a touch more clarity. Finally I think the match statement actually reduces to columnMapping.getOrElse(field.name, field.name)?

[srowen]

@cloud-fan @marmbrus @rxin could we maybe get an opinion on the API change? it's the idea of adding a column mapping to allow for the dataframe's column names to map to different names in a table. I hesitate to commit this without a thumbs up from someone else.

[rxin]

Wouldn't a better solution is to just use DataFrame's own renaming of columns? I am not sure why we want this to be a jdbc specific features. If we do this, don't we need to do it for every single other data source?

[srowen]

Yeah fair point -- just project / rename a dataframe first. WDYT @CK50 ?
It doesn't change the underlying point about sometimes needing to explicitly name the columns in the INSERT.

[rxin]

Explicit naming columns in inserts is ok (good to have actually) - but I wouldn't add a jdbc specific column naming.

[marmbrus]

+1 to explicit naming but without the extra jdbc specific ability to rename

[CK50]

@srowen, @rxin, @marmbrus
If everyone is fine with always using column names in the generated INSERT statement and that these column names will always be taken from the DataFrame object, then I can go back to my earlier approach.
Can you please confirm? If so, I will change code accordingly.

[srowen]

@CK50 yes I personally think it's safer and more compatible to always put col names in the INSERT statement, and assume the caller has lined up the DataFrame column names. So yes it's your same change but just with the mapping logic backed out.

[CK50]

Closing and going back to initial approach in new PR