Commit 2503375
authored
Remove unnecessary allocation in ArrowStreamWriter (#73)
## What's Changed
ArrowStreamWriter allocates 8k of memory by creating a new array pool.
This change introduces a shared buffer instead of pool to reduce the
allocations.
The array pool is used to rent small arrays (8 bytes) but the pool
allocates much bigger arrays (8kb in total)
The Array pool has access time overhead for small arrays compared to
direct allocation.
Results from benchmarks:
Old implementation:
| Method | BatchLength | ColumnSetCount | Mean | Error | StdDev |
Allocated |
|----------- |------------ |---------------
|-----------:|----------:|----------:|----------:|
| WriteBatch | 10000 | 10 | 6.118 ms | 0.1215 ms | 0.3345 ms | 248.53 KB
|
| WriteBatch | 10000 | 14 | 9.788 ms | 0.1910 ms | 0.3396 ms | 324.12 KB
|
| WriteBatch | 300000 | 10 | 119.351 ms | 3.1713 ms | 9.3008 ms | 248.53
KB |
| WriteBatch | 300000 | 14 | 136.697 ms | 2.9229 ms | 8.4799 ms | 324.12
KB |
New implementation:
| Method | BatchLength | ColumnSetCount | Mean | Error | StdDev | Median
| Allocated |
|----------- |------------ |---------------
|-----------:|----------:|-----------:|-----------:|----------:|
| WriteBatch | 10000 | 10 | 5.925 ms | 0.2057 ms | 0.6001 ms | 5.843 ms
| 240.64 KB |
| WriteBatch | 10000 | 14 | 8.908 ms | 0.2743 ms | 0.8002 ms | 8.778 ms
| 316.23 KB |
| WriteBatch | 300000 | 10 | 94.835 ms | 1.7872 ms | 3.7699 ms | 93.892
ms | 240.64 KB |
| WriteBatch | 300000 | 14 | 147.995 ms | 3.6873 ms | 10.6975 ms |
144.591 ms | 316.23 KB |
Closes #41.1 parent fbbe78e commit 2503375
2 files changed
+5
-7
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
14 | 14 | | |
15 | 15 | | |
16 | 16 | | |
| 17 | + | |
17 | 18 | | |
18 | 19 | | |
19 | 20 | | |
| |||
221 | 222 | | |
222 | 223 | | |
223 | 224 | | |
224 | | - | |
| 225 | + | |
225 | 226 | | |
226 | 227 | | |
227 | 228 | | |
| |||
292 | 293 | | |
293 | 294 | | |
294 | 295 | | |
295 | | - | |
| 296 | + | |
296 | 297 | | |
297 | 298 | | |
298 | 299 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
611 | 611 | | |
612 | 612 | | |
613 | 613 | | |
614 | | - | |
615 | | - | |
616 | 614 | | |
617 | 615 | | |
618 | 616 | | |
| |||
663 | 661 | | |
664 | 662 | | |
665 | 663 | | |
666 | | - | |
667 | 664 | | |
668 | 665 | | |
669 | 666 | | |
| |||
1277 | 1274 | | |
1278 | 1275 | | |
1279 | 1276 | | |
1280 | | - | |
| 1277 | + | |
1281 | 1278 | | |
1282 | 1279 | | |
1283 | 1280 | | |
| |||
1294 | 1291 | | |
1295 | 1292 | | |
1296 | 1293 | | |
1297 | | - | |
| 1294 | + | |
1298 | 1295 | | |
1299 | 1296 | | |
1300 | 1297 | | |
| |||
0 commit comments