Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions go/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ The four bindings in this repository are released together and give one answer,
### Changed

- `Schema.Rows` decodes its result with far fewer allocations, so it scales across goroutines (#456). It reads the batch document in one streaming pass into typed values, with no generic tree and no per-row path string, and falls back to the strict generic reader for any document it will not judge itself (an unknown key, a duplicate key, a value of the wrong type), so every refusal and its message are unchanged. On a real 100-row document the decode went from 4,476 to 173 allocations at `doc_flags` 0 and from 56,822 to 2,641 at `doc_flags` 7. The decoded values are identical, and no exported name changes.
- `Schema.Rows` makes far fewer allocations around the decode too (#456). Measured on a full call over a 100-row body with a filter, a four-key settings map and an export (the shape of the report), it went from 412 allocations and 75 KB per call to 76 and 45 KB, about 40% fewer bytes for the collector to clear. The batch document is copied out of the library into a pooled buffer instead of a fresh slice, the row decoders and the settings encoder are kept between calls, the per-row verdict, partition id and span pointers come from shared chunks, a list of spans is decoded one element at a time with no allocation each, and the call options are small named types instead of closures. Every returned value is still Go-owned and independent of those pools, and results are byte-for-byte what they were; no exported name changes.

### Fixed

Expand Down
19 changes: 19 additions & 0 deletions go/chtypes/capture_batch_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -64,4 +64,23 @@ func TestCaptureBatchDocuments(t *testing.T) {
}
t.Logf("wrote %s (%d bytes)", name, len(doc))
}
// The third document is the shape a filtering caller gets: a filter
// attached, so the document also carries rows_passed, rows_cut and one
// row_spans entry per row, and an export asked for.
flt, cerr := sch.lib.tbl.FilterCreate(sch.h, []byte("small < {m:Int8}"), []byte(`{"m":"50"}`), nil)
if cerr != nil {
t.Fatal(callError(cerr))
}
defer flt.Close()
docBuf, exportBuf, cerr := sch.lib.tbl.PreviewBatch(sch.h, int32(JSONEachRow), body, nil, nil, flt, int32(JSONCompactEachRow), 0)
if cerr != nil {
t.Fatal(callError(cerr))
}
doc := sch.lib.tbl.Take(docBuf)
sch.lib.tbl.Take(exportBuf)
name := filepath.Join(dir, "batch-100-filter.json")
if err := os.WriteFile(name, doc, 0o644); err != nil {
t.Fatal(err)
}
t.Logf("wrote %s (%d bytes)", name, len(doc))
}
Loading
Loading