-
Notifications
You must be signed in to change notification settings - Fork 1.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Use hardware accelerated CRC32C function if available #1389
Conversation
Oh interesting. Two thoughts:
|
@jeffwidman Yea, will patch the docs. |
@@ -70,3 +70,16 @@ Install the `python-snappy` module | |||
.. code:: bash | |||
|
|||
pip install python-snappy | |||
|
|||
|
|||
Optional crc32c install |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@jeffwidman Could you read this through, good enough?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks great
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM, I held off on merging this @tvoinarovskyi in case you want to make any more changes first... when you're ready please press the green button.
@@ -70,3 +70,16 @@ Install the `python-snappy` module | |||
.. code:: bash | |||
|
|||
pip install python-snappy | |||
|
|||
|
|||
Optional crc32c install |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks great
@dpkp @jeffwidman Hehe, I'm an idiot for this =) So, here I was pushing my butt to accelerate varint implementation, but did not look properly at the profiler.
Long story short, while accelerating the V2 implementation for
aiokafka
I found that_crc32c.py
is super slow and takes more time for 1 call than 100append
. So as a fast fix I propose to use https://github.com/ICRAR/crc32c if available on the system. I will look into_crc32c.py
script later to make it a bit faster.Benchmark without HW crc32:
With HW crc32:
So now V2 is relatively the same speed as V1 and V0. Now we can move on to TransactionalProducer =)