append with bulk insert does results in NoSuchMethodError with DB runtime 12.2 LTS while overwrite works

[eschnepel]

I am inserting data into a Azure SQL DB table using bulk inserts with mode "append".

When the table does not exist before the table is created as expected.
When the table does already exist I am getting the following error:

An error occurred while calling o6686.save.
: java.lang.NoSuchMethodError: org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$.getSchema$default$4()Z
at com.microsoft.sqlserver.jdbc.spark.BulkCopyUtils$.matchSchemas(BulkCopyUtils.scala:305)
at com.microsoft.sqlserver.jdbc.spark.BulkCopyUtils$.getColMetaData(BulkCopyUtils.scala:266)
at com.microsoft.sqlserver.jdbc.spark.Connector.write(Connector.scala:79)
....

some searching around, I found the commit
apache/spark@7021806#diff-c3859e97335ead4b131263565c987d877bea0af3adbd6c5bf2d3716768d2e083
which changed the interface for the getSchema function. But the commit relates only to the master branch is tagged for release 4.0.0.

[ANSHUL22122001]

Did you get any solution for this?
we are also facing the same issue, when we upgraded pyspark version to 3.4

[ANSHUL22122001]

Did you get any solution for this?
we are also facing the same issue, when we upgraded pyspark version to 3.4