-
Notifications
You must be signed in to change notification settings - Fork 75
Description
Thanks for your library.
Because you can't update a struct dynamically in go, when scanning a row into a struct there isn't much reason to create a columnToFieldIndex map for the same struct more than once.
I created a test struct with 1024 fields and some quick benchmarks to see if the cache helps, you can see the benchmarks in my fork of scanny here.
To implement a cache I only changed the getColumnToFieldIndexMap
function which will attempt to get a column to field index map from and return it if it exists. The cache can be implemented using both a map[reflect.Type]
with a sync.RWMutex
or with a sync.Map
. I tested both in the benchmarks.
The BenchmarkStruct functions reuse the same row scanner for each iteration. The BenchmarkScannerStruct functions create a new row scanner for each iteration. The benchmarks that end in MapCache use a map[reflect.Type]
with a sync.RWMutex
, while the SyncMapCache benchmarks use a sync.Map
to store the column to field index maps. The results of the benchmarks:
goos: darwin
goarch: amd64
pkg: github.com/georgysavva/scany
BenchmarkStruct
BenchmarkStruct-8 14983 78204 ns/op 16396 B/op 1 allocs/op
BenchmarkStruct_MapCache
BenchmarkStruct_MapCache-8 14470 88521 ns/op 16397 B/op 1 allocs/op
BenchmarkStruct_SyncMapCache
BenchmarkStruct_SyncMapCache-8 14360 80351 ns/op 16397 B/op 1 allocs/op
BenchmarkScannerStruct
BenchmarkScannerStruct-8 2630 462315 ns/op 188686 B/op 3081 allocs/op
BenchmarkScannerStruct_MapCache
BenchmarkScannerStruct_MapCache-8 7016 147001 ns/op 57477 B/op 4 allocs/op
BenchmarkScannerStruct_SyncMapCache
BenchmarkScannerStruct_SyncMapCache-8 7268 149239 ns/op 57476 B/op 4 allocs/op
BenchmarkMap
BenchmarkMap-8 5004 246030 ns/op 114842 B/op 2054 allocs/op
PASS
When reusing a row scanner there isn't much difference to the performance when using a cache. The real benefit happens when you create a new row scanner each iteration. The allocs drop from 3081 to only 4. And both the bytes/op and ns/op drop by over three times.
I think this would be a useful feature to add since even though having 1024 fields in a struct is pretty extreme but from the benchmarks the getColumnToFieldIndexMap
function is creating 3 allocs per field of a struct which can be avoided after the first call to getColumnToFieldIndexMap
with the same struct without much overhead.