Skip to content

Performance enhancement by caching fastavro schema parsed on every message serialized & deserialized #627

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Aug 7, 2019

Conversation

bruc3mackenzi3
Copy link

The fastavro schema is being re-parsed on each message produced, which is a detriment to performance. This PR addresses it by caching the parsed fastavro schema instead of the JSON dict schema.

We see a 25% performance enhancement when used with large schemas, and a 5% improvement with small schemas.

@ghost
Copy link

ghost commented Jun 19, 2019

It looks like @bruc3mackenzi3 hasn't signed our Contributor License Agreement, yet.

The purpose of a CLA is to ensure that the guardian of a project's outputs has the necessary ownership or grants of rights over all contributions to allow them to distribute under the chosen licence.
Wikipedia

You can read and sign our full Contributor License Agreement here.

Once you've signed reply with [clabot:check] to prove it.

Appreciation of efforts,

clabot

@bruc3mackenzi3
Copy link
Author

[clabot:check]

@ghost
Copy link

ghost commented Jun 19, 2019

@confluentinc It looks like @bruc3mackenzi3 just signed our Contributor License Agreement. 👍

Always at your service,

clabot

@edenhill edenhill requested a review from rnpridgeon June 19, 2019 15:24
@AndreiLieb
Copy link

@rnpridgeon any updates on this? This is a pretty significant optimization for us.

@rnpridgeon
Copy link
Contributor

Good catch @bruc3mackenzi3 , all in all the PR looks good. I'll sanity check it against the Avro integration tests this afternoon for both the py2 and py3 interpreters and if everything works as expected I'll merge. Thanks again for your contribution and I apologize for the hold up on this review.

@bruc3mackenzi3
Copy link
Author

@rnpridgeon thanks for taking a look and that's great news. We look forward to having our PR merged. If you need anything from us in the meantime let us know.

@rnpridgeon rnpridgeon merged commit 50e09cf into confluentinc:master Aug 7, 2019
@rnpridgeon
Copy link
Contributor

Thanks for your contribution @bruc3mackenzi3

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants