fix: MixedBatchWriter should nil the slice instead of zeroing

[disq]

Helps with memory.

Benched with #1552, old:

BenchmarkWriterMemory/MixedBatchWriter_batch10k_bytes100M
BenchmarkWriterMemory/MixedBatchWriter_batch10k_bytes100M-10    	     826	   1445595 ns/op	      1393 bytes/op
BenchmarkWriterMemory/MixedBatchWriter_defaults
BenchmarkWriterMemory/MixedBatchWriter_defaults-10              	     670	   1504799 ns/op	      1393 bytes/op
BenchmarkWriterMemory/MixedBatchWriter_wide_batch10k_bytes100M
BenchmarkWriterMemory/MixedBatchWriter_wide_batch10k_bytes100M-10         	       8	 128966609 ns/op	    172785 bytes/op
BenchmarkWriterMemory/MixedBatchWriter_wide_defaults
BenchmarkWriterMemory/MixedBatchWriter_wide_defaults-10                   	       9	 118630315 ns/op	     60355 bytes/op

new:

BenchmarkWriterMemory/MixedBatchWriter_batch10k_bytes100M
BenchmarkWriterMemory/MixedBatchWriter_batch10k_bytes100M-10    	     690	   1476479 ns/op	      1401 bytes/op
BenchmarkWriterMemory/MixedBatchWriter_defaults
BenchmarkWriterMemory/MixedBatchWriter_defaults-10              	     687	   1539707 ns/op	      1401 bytes/op
BenchmarkWriterMemory/MixedBatchWriter_wide_batch10k_bytes100M
BenchmarkWriterMemory/MixedBatchWriter_wide_batch10k_bytes100M-10         	       8	 129154755 ns/op	    173592 bytes/op
BenchmarkWriterMemory/MixedBatchWriter_wide_defaults
BenchmarkWriterMemory/MixedBatchWriter_wide_defaults-10                   	       9	 111416773 ns/op	     44588 bytes/op

[github-actions]

⏱️ Benchmark results

• Glob-8 ns/op: 92.42

[candiduslynx]

1. When you use nil you immediately introduce a bug with cap (see: https://github.com/cloudquery/plugin-sdk/blob/main/writers/mixedbatchwriter/mixedbatchwriter.go#L177, https://github.com/cloudquery/plugin-sdk/blob/main/writers/mixedbatchwriter/mixedbatchwriter.go#L209)
2. I don't see much improvements here either, maybe I'm not looking correctly

nil

BenchmarkWriterMemory/MixedBatchWriter_batch10k_bytes100M-10                1000              2724 ns/op              1721 bytes/op         1721 B/op         23 allocs/op
BenchmarkWriterMemory/MixedBatchWriter_batch10k_bytes100M-10                1000              2463 ns/op              1721 bytes/op         1721 B/op         23 allocs/op
BenchmarkWriterMemory/MixedBatchWriter_batch10k_bytes100M-10                1000              2053 ns/op              1721 bytes/op         1721 B/op         23 allocs/op
BenchmarkWriterMemory/MixedBatchWriter_batch10k_bytes100M-10                1000              1732 ns/op              1721 bytes/op         1721 B/op         23 allocs/op
BenchmarkWriterMemory/MixedBatchWriter_batch10k_bytes100M-10                1000              1639 ns/op              1721 bytes/op         1721 B/op         23 allocs/op
BenchmarkWriterMemory/MixedBatchWriter_batch10k_bytes100M-10                1000              1802 ns/op              1721 bytes/op         1721 B/op         23 allocs/op
BenchmarkWriterMemory/MixedBatchWriter_batch10k_bytes100M-10                1000              1660 ns/op              1721 bytes/op         1721 B/op         23 allocs/op
BenchmarkWriterMemory/MixedBatchWriter_batch10k_bytes100M-10                1000              1673 ns/op              1393 bytes/op         1393 B/op         23 allocs/op
BenchmarkWriterMemory/MixedBatchWriter_batch10k_bytes100M-10                1000              1739 ns/op              1721 bytes/op         1721 B/op         23 allocs/op
BenchmarkWriterMemory/MixedBatchWriter_batch10k_bytes100M-10                1000              1722 ns/op              1721 bytes/op         1720 B/op         23 allocs/op
BenchmarkWriterMemory/MixedBatchWriter_defaults-10                          1000              1760 ns/op              1721 bytes/op         1721 B/op         23 allocs/op
BenchmarkWriterMemory/MixedBatchWriter_defaults-10                          1000              1646 ns/op              1721 bytes/op         1721 B/op         23 allocs/op
BenchmarkWriterMemory/MixedBatchWriter_defaults-10                          1000              1472 ns/op              1721 bytes/op         1720 B/op         23 allocs/op
BenchmarkWriterMemory/MixedBatchWriter_defaults-10                          1000              1618 ns/op              1721 bytes/op         1721 B/op         23 allocs/op
BenchmarkWriterMemory/MixedBatchWriter_defaults-10                          1000              1692 ns/op              1721 bytes/op         1721 B/op         23 allocs/op
BenchmarkWriterMemory/MixedBatchWriter_defaults-10                          1000              1748 ns/op              1721 bytes/op         1721 B/op         23 allocs/op
BenchmarkWriterMemory/MixedBatchWriter_defaults-10                          1000              1870 ns/op              1721 bytes/op         1720 B/op         23 allocs/op
BenchmarkWriterMemory/MixedBatchWriter_defaults-10                          1000              1796 ns/op              1721 bytes/op         1721 B/op         23 allocs/op
BenchmarkWriterMemory/MixedBatchWriter_defaults-10                          1000              1574 ns/op              1721 bytes/op         1721 B/op         23 allocs/op
BenchmarkWriterMemory/MixedBatchWriter_defaults-10                          1000              1828 ns/op              1721 bytes/op         1721 B/op         23 allocs/op
BenchmarkWriterMemory/MixedBatchWriter_wide_batch10k_bytes100M-10           1000            124192 ns/op            174793 bytes/op       232125 B/op       2049 allocs/op
BenchmarkWriterMemory/MixedBatchWriter_wide_batch10k_bytes100M-10           1000            121860 ns/op            183238 bytes/op       232116 B/op       2049 allocs/op
BenchmarkWriterMemory/MixedBatchWriter_wide_batch10k_bytes100M-10           1000            126347 ns/op            175911 bytes/op       232126 B/op       2049 allocs/op
BenchmarkWriterMemory/MixedBatchWriter_wide_batch10k_bytes100M-10           1000            122443 ns/op            180146 bytes/op       232120 B/op       2049 allocs/op
BenchmarkWriterMemory/MixedBatchWriter_wide_batch10k_bytes100M-10           1000            121118 ns/op            180755 bytes/op       232108 B/op       2049 allocs/op
BenchmarkWriterMemory/MixedBatchWriter_wide_batch10k_bytes100M-10           1000            125056 ns/op            183281 bytes/op       232103 B/op       2049 allocs/op
BenchmarkWriterMemory/MixedBatchWriter_wide_batch10k_bytes100M-10           1000            123030 ns/op            186821 bytes/op       232120 B/op       2049 allocs/op
BenchmarkWriterMemory/MixedBatchWriter_wide_batch10k_bytes100M-10           1000            121279 ns/op            178714 bytes/op       232097 B/op       2049 allocs/op
BenchmarkWriterMemory/MixedBatchWriter_wide_batch10k_bytes100M-10           1000            124958 ns/op            178596 bytes/op       232105 B/op       2049 allocs/op
BenchmarkWriterMemory/MixedBatchWriter_wide_batch10k_bytes100M-10           1000            125754 ns/op            183995 bytes/op       232115 B/op       2049 allocs/op
BenchmarkWriterMemory/MixedBatchWriter_wide_defaults-10                     1000            123916 ns/op            173773 bytes/op       232106 B/op       2049 allocs/op
BenchmarkWriterMemory/MixedBatchWriter_wide_defaults-10                     1000            125387 ns/op            185542 bytes/op       232122 B/op       2049 allocs/op
BenchmarkWriterMemory/MixedBatchWriter_wide_defaults-10                     1000            122312 ns/op            181158 bytes/op       232114 B/op       2049 allocs/op
BenchmarkWriterMemory/MixedBatchWriter_wide_defaults-10                     1000            122467 ns/op            181971 bytes/op       232112 B/op       2049 allocs/op
BenchmarkWriterMemory/MixedBatchWriter_wide_defaults-10                     1000            125174 ns/op            183753 bytes/op       232113 B/op       2049 allocs/op
BenchmarkWriterMemory/MixedBatchWriter_wide_defaults-10                     1000            122004 ns/op            180405 bytes/op       232121 B/op       2049 allocs/op
BenchmarkWriterMemory/MixedBatchWriter_wide_defaults-10                     1000            123294 ns/op            176132 bytes/op       232100 B/op       2049 allocs/op
BenchmarkWriterMemory/MixedBatchWriter_wide_defaults-10                     1000            124500 ns/op            180189 bytes/op       232115 B/op       2049 allocs/op
BenchmarkWriterMemory/MixedBatchWriter_wide_defaults-10                     1000            127433 ns/op            174755 bytes/op       232112 B/op       2049 allocs/op
BenchmarkWriterMemory/MixedBatchWriter_wide_defaults-10                     1000            123012 ns/op            176072 bytes/op       232121 B/op       2049 allocs/op

[:0]

BenchmarkWriterMemory/MixedBatchWriter_batch10k_bytes100M-10                1000              2776 ns/op              1721 bytes/op         1721 B/op         23 allocs/op
BenchmarkWriterMemory/MixedBatchWriter_batch10k_bytes100M-10                1000              2582 ns/op              1721 bytes/op         1721 B/op         23 allocs/op
BenchmarkWriterMemory/MixedBatchWriter_batch10k_bytes100M-10                1000              2208 ns/op              1721 bytes/op         1721 B/op         23 allocs/op
BenchmarkWriterMemory/MixedBatchWriter_batch10k_bytes100M-10                1000              1536 ns/op              1721 bytes/op         1721 B/op         23 allocs/op
BenchmarkWriterMemory/MixedBatchWriter_batch10k_bytes100M-10                1000              1614 ns/op              1721 bytes/op         1721 B/op         23 allocs/op
BenchmarkWriterMemory/MixedBatchWriter_batch10k_bytes100M-10                1000              1646 ns/op              1721 bytes/op         1721 B/op         23 allocs/op
BenchmarkWriterMemory/MixedBatchWriter_batch10k_bytes100M-10                1000              1754 ns/op              1721 bytes/op         1721 B/op         23 allocs/op
BenchmarkWriterMemory/MixedBatchWriter_batch10k_bytes100M-10                1000              1702 ns/op              1721 bytes/op         1721 B/op         23 allocs/op
BenchmarkWriterMemory/MixedBatchWriter_batch10k_bytes100M-10                1000              1724 ns/op              1393 bytes/op         1393 B/op         23 allocs/op
BenchmarkWriterMemory/MixedBatchWriter_batch10k_bytes100M-10                1000              1762 ns/op              1721 bytes/op         1721 B/op         23 allocs/op
BenchmarkWriterMemory/MixedBatchWriter_defaults-10                          1000              1834 ns/op              1721 bytes/op         1721 B/op         23 allocs/op
BenchmarkWriterMemory/MixedBatchWriter_defaults-10                          1000              1693 ns/op              1721 bytes/op         1720 B/op         23 allocs/op
BenchmarkWriterMemory/MixedBatchWriter_defaults-10                          1000              1804 ns/op              1721 bytes/op         1720 B/op         23 allocs/op
BenchmarkWriterMemory/MixedBatchWriter_defaults-10                          1000              1797 ns/op              1721 bytes/op         1720 B/op         23 allocs/op
BenchmarkWriterMemory/MixedBatchWriter_defaults-10                          1000              1739 ns/op              1721 bytes/op         1720 B/op         23 allocs/op
BenchmarkWriterMemory/MixedBatchWriter_defaults-10                          1000              1832 ns/op              1721 bytes/op         1721 B/op         23 allocs/op
BenchmarkWriterMemory/MixedBatchWriter_defaults-10                          1000              1727 ns/op              1721 bytes/op         1721 B/op         23 allocs/op
BenchmarkWriterMemory/MixedBatchWriter_defaults-10                          1000              1700 ns/op              1721 bytes/op         1721 B/op         23 allocs/op
BenchmarkWriterMemory/MixedBatchWriter_defaults-10                          1000              1726 ns/op              1721 bytes/op         1721 B/op         23 allocs/op
BenchmarkWriterMemory/MixedBatchWriter_defaults-10                          1000              1771 ns/op              1721 bytes/op         1721 B/op         23 allocs/op
BenchmarkWriterMemory/MixedBatchWriter_wide_batch10k_bytes100M-10           1000            123076 ns/op            179582 bytes/op       232118 B/op       2049 allocs/op
BenchmarkWriterMemory/MixedBatchWriter_wide_batch10k_bytes100M-10           1000            124413 ns/op            177396 bytes/op       232121 B/op       2049 allocs/op
BenchmarkWriterMemory/MixedBatchWriter_wide_batch10k_bytes100M-10           1000            122790 ns/op            179931 bytes/op       232106 B/op       2049 allocs/op
BenchmarkWriterMemory/MixedBatchWriter_wide_batch10k_bytes100M-10           1000            121242 ns/op            182132 bytes/op       232107 B/op       2049 allocs/op
BenchmarkWriterMemory/MixedBatchWriter_wide_batch10k_bytes100M-10           1000            120752 ns/op            182754 bytes/op       232109 B/op       2049 allocs/op
BenchmarkWriterMemory/MixedBatchWriter_wide_batch10k_bytes100M-10           1000            122812 ns/op            181840 bytes/op       232118 B/op       2049 allocs/op
BenchmarkWriterMemory/MixedBatchWriter_wide_batch10k_bytes100M-10           1000            123734 ns/op            176544 bytes/op       232118 B/op       2049 allocs/op
BenchmarkWriterMemory/MixedBatchWriter_wide_batch10k_bytes100M-10           1000            124438 ns/op            181274 bytes/op       232122 B/op       2049 allocs/op
BenchmarkWriterMemory/MixedBatchWriter_wide_batch10k_bytes100M-10           1000            123256 ns/op            182356 bytes/op       231774 B/op       2049 allocs/op
BenchmarkWriterMemory/MixedBatchWriter_wide_batch10k_bytes100M-10           1000            122667 ns/op            178061 bytes/op       232104 B/op       2049 allocs/op
BenchmarkWriterMemory/MixedBatchWriter_wide_defaults-10                     1000            122845 ns/op            182142 bytes/op       232100 B/op       2049 allocs/op
BenchmarkWriterMemory/MixedBatchWriter_wide_defaults-10                     1000            123886 ns/op            182779 bytes/op       232111 B/op       2049 allocs/op
BenchmarkWriterMemory/MixedBatchWriter_wide_defaults-10                     1000            123729 ns/op            182779 bytes/op       232096 B/op       2049 allocs/op
BenchmarkWriterMemory/MixedBatchWriter_wide_defaults-10                     1000            123632 ns/op            175236 bytes/op       231790 B/op       2049 allocs/op
BenchmarkWriterMemory/MixedBatchWriter_wide_defaults-10                     1000            122319 ns/op            181572 bytes/op       232103 B/op       2049 allocs/op
BenchmarkWriterMemory/MixedBatchWriter_wide_defaults-10                     1000            132980 ns/op            179910 bytes/op       232122 B/op       2049 allocs/op
BenchmarkWriterMemory/MixedBatchWriter_wide_defaults-10                     1000            124032 ns/op            177858 bytes/op       232121 B/op       2049 allocs/op
BenchmarkWriterMemory/MixedBatchWriter_wide_defaults-10                     1000            124370 ns/op            181475 bytes/op       232089 B/op       2049 allocs/op
BenchmarkWriterMemory/MixedBatchWriter_wide_defaults-10                     1000            124490 ns/op            179209 bytes/op       232124 B/op       2049 allocs/op
BenchmarkWriterMemory/MixedBatchWriter_wide_defaults-10                     1000            119278 ns/op            182648 bytes/op       232124 B/op       2049 allocs/op

[candiduslynx]

Yeah, seems like pg v7.3.7 (without this change) can write 8K rows/sec in the AWS src scenario, while 7.3.8 (with this fix) can handle write only around 3K rows/sec

[disq]

Looks like this introduced a bug regarding cap since we don't have the set batch size capacity any more and the memory "improvements" might be related to that. In this case the batches would be flushed when the element size reaches 2^n (possibly 1, but if we don't check/call flush all the time it can go up in 2^n increments, 1, 2, 4, 8, ...)