You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
SUM/AVG results differ between 1 thread and 12 threads #185
SUM and AVG return different values depending on thread count, because the parallel path sums per-worker partials in a different order than the sequential path sums rows. The difference is in the last digits, but it means the same query on the same file gives two different answers depending on --threads.
Repro
2M row file, 8 groups:
csvql "SELECT country, SUM(amount), AVG(amount) FROM 'small.csv' GROUP BY country"
BR,125002500.00000001,500.01000000000005
csvql --threads 1 "SELECT country, SUM(amount), AVG(amount) FROM 'small.csv' GROUP BY country"
BR,125002500,500.01
duckdb "SELECT country, SUM(amount), AVG(amount) FROM read_csv('small.csv',header=true) GROUP BY country"
BR,125002500.0,500.01
The single-threaded value matches DuckDB. The parallel one does not.
Not a regression
Confirmed present on main before #180/#182/#183/#184 by rebuilding the prior commit and running the same two queries — output was byte-identical to the above. This is long-standing, not introduced by the recent aggregation work.
Why it happens
Each worker keeps a running f64 sum for its chunk and the merge adds those partials together. Floating point addition is not associative, so partial-then-combine gives a different result than accumulating every row in file order. Worker chunk boundaries depend on thread count, so the answer moves with --threads.
Notes for whoever picks this up
It also makes --threads a correctness-visible flag, which is worth deciding on deliberately.
Neumaier/Kahan compensated summation in the accumulator and in the merge would make both paths agree and match DuckDB, at some cost per row.
Worth checking whether VARIANCE/STDDEV (which use sum_sqs) drift further, since squaring amplifies it.
MIN/MAX/COUNT are order-independent and unaffected.
Activity
docs: blog post on the five-way distinct benchmark and the bug it fou…
fix: SUM/AVG no longer change with --threads (#185) (#198)