Commits on Aug 22, 2023

1. [SPARK-44871][SQL] Fix percentile_disc behaviour …

   This PR fixes `percentile_disc()` function as currently it returns inforrect results in some cases. E.g.:
   ```
   SELECT
     percentile_disc(0.0) WITHIN GROUP (ORDER BY a) as p0,
     percentile_disc(0.1) WITHIN GROUP (ORDER BY a) as p1,
     percentile_disc(0.2) WITHIN GROUP (ORDER BY a) as p2,
     percentile_disc(0.3) WITHIN GROUP (ORDER BY a) as p3,
     percentile_disc(0.4) WITHIN GROUP (ORDER BY a) as p4,
     percentile_disc(0.5) WITHIN GROUP (ORDER BY a) as p5,
     percentile_disc(0.6) WITHIN GROUP (ORDER BY a) as p6,
     percentile_disc(0.7) WITHIN GROUP (ORDER BY a) as p7,
     percentile_disc(0.8) WITHIN GROUP (ORDER BY a) as p8,
     percentile_disc(0.9) WITHIN GROUP (ORDER BY a) as p9,
     percentile_disc(1.0) WITHIN GROUP (ORDER BY a) as p10
   FROM VALUES (0), (1), (2), (3), (4) AS v(a)
   ```
   currently returns:
   ```
   +---+---+---+---+---+---+---+---+---+---+---+
   | p0| p1| p2| p3| p4| p5| p6| p7| p8| p9|p10|
   +---+---+---+---+---+---+---+---+---+---+---+
   |0.0|0.0|0.0|1.0|1.0|2.0|2.0|2.0|3.0|3.0|4.0|
   +---+---+---+---+---+---+---+---+---+---+---+
   ```
   but after this PR it returns the correct:
   ```
   +---+---+---+---+---+---+---+---+---+---+---+
   | p0| p1| p2| p3| p4| p5| p6| p7| p8| p9|p10|
   +---+---+---+---+---+---+---+---+---+---+---+
   |0.0|0.0|0.0|1.0|1.0|2.0|2.0|3.0|3.0|4.0|4.0|
   +---+---+---+---+---+---+---+---+---+---+---+
   ```
   
   Bugfix.
   
   Yes, fixes a correctness bug, but the old behaviour can be restored with `spark.sql.legacy.percentileDiscCalculation=true`.
   
   Added new UTs.
   
   Closes apache#42559 from peter-toth/SPARK-44871-fix-percentile-disc-behaviour.
   
   Authored-by: Peter Toth <peter.toth@gmail.com>
   Signed-off-by: Peter Toth <peter.toth@gmail.com>

   

   peter-toth committed Aug 22, 2023
   Configuration menu

   • View commit details
   • Copy full SHA for d666a6b
   • Browse repository at this point

   Copy the full SHA

   d666a6b View commit details

   Browse the repository at this point in the history