Skip to content

Feature: cache the columnToFieldIndex map for each struct type #25

@kovalromank

Description

@kovalromank

Thanks for your library.

Because you can't update a struct dynamically in go, when scanning a row into a struct there isn't much reason to create a columnToFieldIndex map for the same struct more than once.

I created a test struct with 1024 fields and some quick benchmarks to see if the cache helps, you can see the benchmarks in my fork of scanny here.

To implement a cache I only changed the getColumnToFieldIndexMap function which will attempt to get a column to field index map from and return it if it exists. The cache can be implemented using both a map[reflect.Type] with a sync.RWMutex or with a sync.Map. I tested both in the benchmarks.

The BenchmarkStruct functions reuse the same row scanner for each iteration. The BenchmarkScannerStruct functions create a new row scanner for each iteration. The benchmarks that end in MapCache use a map[reflect.Type] with a sync.RWMutex, while the SyncMapCache benchmarks use a sync.Map to store the column to field index maps. The results of the benchmarks:

goos: darwin
goarch: amd64
pkg: github.com/georgysavva/scany
BenchmarkStruct
BenchmarkStruct-8                       	   14983	     78204 ns/op	   16396 B/op	       1 allocs/op
BenchmarkStruct_MapCache
BenchmarkStruct_MapCache-8              	   14470	     88521 ns/op	   16397 B/op	       1 allocs/op
BenchmarkStruct_SyncMapCache
BenchmarkStruct_SyncMapCache-8          	   14360	     80351 ns/op	   16397 B/op	       1 allocs/op
BenchmarkScannerStruct
BenchmarkScannerStruct-8                	    2630	    462315 ns/op	  188686 B/op	    3081 allocs/op
BenchmarkScannerStruct_MapCache
BenchmarkScannerStruct_MapCache-8       	    7016	    147001 ns/op	   57477 B/op	       4 allocs/op
BenchmarkScannerStruct_SyncMapCache
BenchmarkScannerStruct_SyncMapCache-8   	    7268	    149239 ns/op	   57476 B/op	       4 allocs/op
BenchmarkMap
BenchmarkMap-8                          	    5004	    246030 ns/op	  114842 B/op	    2054 allocs/op
PASS

When reusing a row scanner there isn't much difference to the performance when using a cache. The real benefit happens when you create a new row scanner each iteration. The allocs drop from 3081 to only 4. And both the bytes/op and ns/op drop by over three times.

I think this would be a useful feature to add since even though having 1024 fields in a struct is pretty extreme but from the benchmarks the getColumnToFieldIndexMap function is creating 3 allocs per field of a struct which can be avoided after the first call to getColumnToFieldIndexMap with the same struct without much overhead.

Metadata

Metadata

Assignees

No one assigned

    Labels

    help wantedExtra attention is needed

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions