Since cd18939 (#3408), flowctl raw preview-next --fixture panics once a connector takes longer than COMPRESS_AFTER (2.5s) per transaction:
thread 'tokio-runtime-worker' panicked at crates/flowctl/src/raw/preview_next/fixture.rs:680:10:
disk_backlog_bytes underflow
ERROR flowctl::raw::preview_next: fixture feeder ended before end-of-stream; stopping active session
Reproduced with tests/benchmark/materialize/run.sh --connector materialize-redshift --scenario large-docs.yaml in estuary/connectors (flowctl v0.6.13-55-gfef96c502c7): the panic lands in round 2, while the connector is busy with a multi-second COPY.
Cause. The fixture charges disk_backlog_bytes with each rolled segment's size. Blocks above the writer's compress threshold are already LZ4-compressed, so when a segment survives COMPRESS_AFTER and SealedSegment::serve compresses the whole file, the file can grow: serve yields old.saturating_sub(new) == 0, sets sealed.size = new_size, and the eventual unlink yields that larger size. on_reclaimed then credits more than it charged and checked_sub(...).expect(...) panics. A consumer faster than 2.5s unlinks before compression and never sees it.
LogActor::on_reclaimed (crates/shuffle/src/log/actor.rs) has the same checked_sub.
Since cd18939 (#3408),
flowctl raw preview-next --fixturepanics once a connector takes longer thanCOMPRESS_AFTER(2.5s) per transaction:Reproduced with
tests/benchmark/materialize/run.sh --connector materialize-redshift --scenario large-docs.yamlin estuary/connectors (flowctl v0.6.13-55-gfef96c502c7): the panic lands in round 2, while the connector is busy with a multi-second COPY.Cause. The fixture charges
disk_backlog_byteswith each rolled segment's size. Blocks above the writer's compress threshold are already LZ4-compressed, so when a segment survivesCOMPRESS_AFTERandSealedSegment::servecompresses the whole file, the file can grow:serveyieldsold.saturating_sub(new) == 0, setssealed.size = new_size, and the eventual unlink yields that larger size.on_reclaimedthen credits more than it charged andchecked_sub(...).expect(...)panics. A consumer faster than 2.5s unlinks before compression and never sees it.LogActor::on_reclaimed(crates/shuffle/src/log/actor.rs) has the samechecked_sub.