-
-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
perf: Refactor CSV serialization to not go thorough AnyValue
#15576
Conversation
AnyValue
AnyValue
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #15576 +/- ##
==========================================
+ Coverage 81.08% 81.19% +0.11%
==========================================
Files 1367 1368 +1
Lines 174872 175289 +417
Branches 2530 2530
==========================================
+ Hits 141799 142333 +534
+ Misses 32598 32482 -116
+ Partials 475 474 -1 ☔ View full report in Codecov by Sentry. |
CodSpeed Performance ReportMerging #15576 will not alter performanceComparing Summary
|
Thanks for the PR. This certainly looks interesting. Will take a look tomorrow. |
While convenient, this harms performance. Avoiding it gives a boost of 40% on my machine, and even more if I limit to single thread. I did several other things to improve performance, like pre-compilation of date/time formatting and speeding up strings with quotes that need to be escaped. Those aren't strictly related, but they are nice.
I went through this carefully and this looks great! 🙌 Really neat monomorphization and constant usage to ensure we get an optimal serializer. Thanks a lot. |
While convenient, this harms performance. Avoiding it gives a boost of 40% on my machine, and even more if I limit to single thread.
I did several other things to improve performance, like pre-compilation of date/time formatting and speeding up strings with quotes that need to be escaped. Those aren't strictly related, but they are nice.
Benchmark (the
POOL.install()
is because otherwise the switch to Polars' pool has a measurable overhead, but this should be only present in benchmarking and not in real-world cases):