Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[improve][client] Add classLoader field for SchemaDefinition #15915

Merged
merged 5 commits into from
Jun 23, 2022

Conversation

coderzc
Copy link
Member

@coderzc coderzc commented Jun 3, 2022

Fixes #15899

Motivation

Now, don‘t register logical type conversions when use SchemaDefinition.<T>builder().withJsonDef() beacase it without classLoader param.

See:

public AvroReader(Schema writerSchema, Schema readerSchema, ClassLoader classLoader,
boolean jsr310ConversionEnabled) {
this.schema = readerSchema;
if (classLoader != null) {
ReflectData reflectData = new ReflectData(classLoader);
AvroSchema.addLogicalTypeConversions(reflectData, jsr310ConversionEnabled);
this.reader = new ReflectDatumReader<>(writerSchema, readerSchema, reflectData);
} else {
this.reader = new ReflectDatumReader<>(writerSchema, readerSchema);
}
}

We can add the classLoader field for SchemaDefinition, user can manually pass a classLoader to register logical type conversions

Modifications

Add classLoader field for SchemaDefinition

Verifying this change

  • Make sure that the change passes the CI checks.

(Please pick either of the following options)

This change added tests and can be verified

Documentation

Check the box below or label this PR directly.

Need to update docs?

  • doc-required
    (Your PR needs to update docs and you will update later)

  • doc-not-needed
    (Please explain why)

  • doc
    (Your PR contains doc changes)

  • doc-complete
    (Docs have been already added)

@github-actions github-actions bot added the doc-not-needed Your PR changes do not impact docs label Jun 3, 2022
@codelipenghui codelipenghui added this to the 2.11.0 milestone Jun 6, 2022
@codelipenghui codelipenghui added release/2.9.3 release/2.10.1 type/bug The PR fixed a bug or issue reported a bug area/schema labels Jun 6, 2022
@codelipenghui
Copy link
Contributor

codelipenghui commented Jun 6, 2022

@coderzc I have created a PR to clean up the code of StructSchema, the removed code is no longer useful when I reviewing this PR.

Copy link
Member

@shibd shibd left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/LGTM

@codelipenghui
Copy link
Contributor

/pulsarbot run-failure-checks

1 similar comment
@mattisonchao
Copy link
Member

/pulsarbot run-failure-checks

@mattisonchao
Copy link
Member

Hi @coderzc

We re-run the test many times, but the test still fails. Could you take a look?

@coderzc
Copy link
Member Author

coderzc commented Jun 9, 2022

Hi @coderzc

We re-run the test many times, but the test still fails. Could you take a look?

ok

@coderzc
Copy link
Member Author

coderzc commented Jun 9, 2022

/pulsarbot run-failure-checks

@coderzc
Copy link
Member Author

coderzc commented Jun 9, 2022

/pulsarbot run-failure-checks

@coderzc coderzc marked this pull request as draft June 10, 2022 00:52
@coderzc coderzc force-pushed the fix_issuse_15899 branch 2 times, most recently from b42e1db to b0dac72 Compare June 10, 2022 11:11
@coderzc coderzc force-pushed the fix_issuse_15899 branch from b0dac72 to 36b723c Compare June 10, 2022 11:13
@coderzc coderzc changed the title [fix][client] Register logical type conversions when use SchemaDefinition.<T>builder().withJsonDef() [fix][client] Add classLoader field for SchemaDefinition Jun 10, 2022
@coderzc coderzc force-pushed the fix_issuse_15899 branch from f112576 to 7325c67 Compare June 10, 2022 13:57
@coderzc coderzc marked this pull request as ready for review June 10, 2022 15:35
*
* @return schema definition builder
*/
SchemaDefinitionBuilder<T> withClassLoader(ClassLoader classLoader);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If need to add a public API to fix the issue, it should start with a proposal first.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, I have been submitted a PIP: #16058

@coderzc
Copy link
Member Author

coderzc commented Jun 23, 2022

@congbobo184 @mattisonchao PTAL

@congbobo184 congbobo184 merged commit 8434500 into apache:master Jun 23, 2022
@coderzc coderzc changed the title [fix][client] Add classLoader field for SchemaDefinition [feature][client] Add classLoader field for SchemaDefinition Jun 24, 2022
@coderzc coderzc changed the title [feature][client] Add classLoader field for SchemaDefinition [improve][client] Add classLoader field for SchemaDefinition Jun 24, 2022
codelipenghui pushed a commit that referenced this pull request Jun 28, 2022
Fixes #15899

### Motivation

Now, don‘t register logical type conversions when use `SchemaDefinition.<T>builder().withJsonDef()` beacase it without  classLoader param.

See:
https://github.com/apache/pulsar/blob/04aa9e8e51869d1621a7e25402a656084eebfc09/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/reader/AvroReader.java#L58-L68

We can add the classLoader field for `SchemaDefinition`, user can manually pass a classLoader to register logical type conversions

### Modifications

Add classLoader field for `SchemaDefinition`

(cherry picked from commit 8434500)
mattisonchao pushed a commit that referenced this pull request Jul 2, 2022
Fixes #15899

### Motivation

Now, don‘t register logical type conversions when use `SchemaDefinition.<T>builder().withJsonDef()` beacase it without  classLoader param.

See:
https://github.com/apache/pulsar/blob/04aa9e8e51869d1621a7e25402a656084eebfc09/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/reader/AvroReader.java#L58-L68

We can add the classLoader field for `SchemaDefinition`, user can manually pass a classLoader to register logical type conversions

### Modifications

Add classLoader field for `SchemaDefinition`

(cherry picked from commit 8434500)
@mattisonchao mattisonchao added the cherry-picked/branch-2.9 Archived: 2.9 is end of life label Jul 2, 2022
nicoloboschi pushed a commit to datastax/pulsar that referenced this pull request Jul 4, 2022
)

Fixes apache#15899

### Motivation

Now, don‘t register logical type conversions when use `SchemaDefinition.<T>builder().withJsonDef()` beacase it without  classLoader param.

See:
https://github.com/apache/pulsar/blob/04aa9e8e51869d1621a7e25402a656084eebfc09/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/reader/AvroReader.java#L58-L68

We can add the classLoader field for `SchemaDefinition`, user can manually pass a classLoader to register logical type conversions

### Modifications

Add classLoader field for `SchemaDefinition`

(cherry picked from commit 8434500)
(cherry picked from commit ea20a89)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/schema cherry-picked/branch-2.9 Archived: 2.9 is end of life cherry-picked/branch-2.10 doc-not-needed Your PR changes do not impact docs release/2.9.4 release/2.10.2 type/bug The PR fixed a bug or issue reported a bug
Projects
None yet
5 participants