Provide Spark 3.1 - 3.5 support #114
Conversation
rozza
requested review from
a team,
jyemin and
stIncMale
and removed request for
a team
|
Apologies - seems to compile fine but there is some more work to do with integration tests. |
| - func: "upload test results" | ||
| - func: "cleanup" | ||
|
|
||
| variables: |
| * @return true if the namespace (database) was dropped | ||
| */ | ||
| @Override | ||
| public boolean dropNamespace(final String[] namespace, final boolean cascade) { |
There was a problem hiding this comment.
dropNamespace(namespace) was removed in 3.4
dropNamespace(namespace, cascade) was added in 3.4 - probably with a default value for cascade.
There was a problem hiding this comment.
Not sure what you mean by a default value for cascade.
There was a problem hiding this comment.
Scala allows default values for parameters - so they may have just done that.
| /** | ||
| * An adapter for SupportsNamespaces as the API for dropNamespace changed between versions | ||
| */ | ||
| interface SupportsNamespacesAdapter { |
There was a problem hiding this comment.
Declare both dropNamespace methods then both can be overridden
There was a problem hiding this comment.
It's crazy that this works! Ingenious.
| throw new ExceptionInInitializerError("Unsupported version of Spark: " + sparkVersion); | ||
| } | ||
|
|
||
| MODULE_STATIC = moduleStatic; |
There was a problem hiding this comment.
Cache the methods that we looked up.
| @Override | ||
| public ExpressionEncoder<Row> apply(final StructType schema) { | ||
| ExpressionEncoder<Row> rowEncoder = MethodInvoker.invoke(APPLY_METHOD, MODULE_STATIC, schema); | ||
| return rowEncoder.resolveAndBind(ATTRIBUTE_FUNCTION.apply(schema), SimpleAnalyzer$.MODULE$); |
There was a problem hiding this comment.
Get the resolved ExpressionEncoder which can create valid serializers and deserializers
|
|
||
| /** Shared converter helper methods and statics */ | ||
| public final class ConverterHelper { | ||
| static final SchemaToExpressionEncoderFunction SCHEMA_TO_EXPRESSION_ENCODER_FUNCTION = |
There was a problem hiding this comment.
Make static the function that is used to convert a schema to an expression encoder.
SPARK-413
|
@jyemin & @stIncMale this is now ready for review :) |
| * @return true if the namespace (database) was dropped | ||
| */ | ||
| @Override | ||
| public boolean dropNamespace(final String[] namespace, final boolean cascade) { |
There was a problem hiding this comment.
Not sure what you mean by a default value for cascade.
| /** | ||
| * An adapter for SupportsNamespaces as the API for dropNamespace changed between versions | ||
| */ | ||
| interface SupportsNamespacesAdapter { |
There was a problem hiding this comment.
It's crazy that this works! Ingenious.
src/main/java/com/mongodb/spark/sql/connector/schema/SchemaToExpressionEncoderFunction.java
Outdated
Show resolved
Hide resolved
src/main/java/com/mongodb/spark/sql/connector/schema/SchemaToExpressionEncoderFunction.java
Outdated
Show resolved
Hide resolved
| /** | ||
| * An adapter for SupportsNamespaces as the API for dropNamespace changed between versions | ||
| */ | ||
| interface SupportsNamespacesAdapter { |
src/main/java/com/mongodb/spark/sql/connector/schema/SchemaToExpressionEncoderFunction.java
Outdated
Show resolved
Hide resolved
3 participants
Two core fixes:
SupportsNamespaceswhich addedcascadetodropNamespace.InternalRowtoRowSPARK-413