Skip to content

SUM/AVG results differ between 1 thread and 12 threads #185

Closed

Description

@melihbirim

SUM and AVG return different values depending on thread count, because the parallel path sums per-worker partials in a different order than the sequential path sums rows. The difference is in the last digits, but it means the same query on the same file gives two different answers depending on --threads.

Repro

2M row file, 8 groups:

csvql "SELECT country, SUM(amount), AVG(amount) FROM 'small.csv' GROUP BY country"
  BR,125002500.00000001,500.01000000000005

csvql --threads 1 "SELECT country, SUM(amount), AVG(amount) FROM 'small.csv' GROUP BY country"
  BR,125002500,500.01

duckdb "SELECT country, SUM(amount), AVG(amount) FROM read_csv('small.csv',header=true) GROUP BY country"
  BR,125002500.0,500.01

The single-threaded value matches DuckDB. The parallel one does not.

Not a regression

Confirmed present on main before #180/#182/#183/#184 by rebuilding the prior commit and running the same two queries — output was byte-identical to the above. This is long-standing, not introduced by the recent aggregation work.

Why it happens

Each worker keeps a running f64 sum for its chunk and the merge adds those partials together. Floating point addition is not associative, so partial-then-combine gives a different result than accumulating every row in file order. Worker chunk boundaries depend on thread count, so the answer moves with --threads.

Notes for whoever picks this up

  • It also makes --threads a correctness-visible flag, which is worth deciding on deliberately.
  • Neumaier/Kahan compensated summation in the accumulator and in the merge would make both paths agree and match DuckDB, at some cost per row.
  • Worth checking whether VARIANCE/STDDEV (which use sum_sqs) drift further, since squaring amplifies it.
  • MIN/MAX/COUNT are order-independent and unaffected.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions