EPIC: Enhance Auron's Correctness Testing

[xumingming]

Background

Auron is a great Native Engine. But we also see some correctness related issues recently:

• Upper function incorrectly mapped to Lower implementation #1674
• string cast array missing null value #1716
• initCap behavior mismatches Spark semantics #1680

Apache Gluten(https://github.com/apache/incubator-gluten) faces the similar issue, Gluten has built a great mechanism that runs all the Unit Test bundled with Spark(https://github.com/apache/incubator-gluten/tree/main/gluten-ut), but with native engine enabled. Doing this Gluten can check whether Gluten produces the same result as Vanilla Spark.

I think Auron can copy Gluten's idea to do the similar thing, to enhance the user's confidence to use Auron in production environment.

Sub Tasks

Adding all the tests of Spark involves a lot work, so we divide them into sub tasks, feel free to take one:

• [Correctness Testing] Spark 3.3 - Add Aggregate operator related tests #1812
• [Correctness Testing] Spark 3.3 - Add Join operator related tests #1813
• [Correctness Testing] Spark 3.3 - Add Sort operator related tests #1814
• [Correctness Testing] Spark 3.3 - Add Exchange operator related tests #1815
• [Correctness Testing] Spark 3.3 - Add Parquet related tests #1816
• [Correctness Testing] Spark 3.3 - Add Functions related tests #1817
• [Correctness Testing] Spark 3.3 - Add Expressions related tests #1818

Correctness Issues We Found

• [Failed] Substring does not support binary input #1724
• [Correctness] ExistenceJoin/LeftAnti + BuildLeft produces incorrect result when probe side is empty #1807
• [Correctness] test single condition (equal) for left anti join using ShuffledHashJoin FAILED #1810
• [Correctness] test single unique condition (equal) for left anti join using ShuffledHashJoin FAILED #1811
• [Failed] inner join, multiple matches using ShuffledHashJoin FAILED #1821

[xumingming]

How do you think? @ShreyeshArangath @cxzl25

[cxzl25]

I think it's a good idea. Now Auron UT is missing a lot, and some investment is needed.

[ShreyeshArangath]

Big +1! I think we should carve out an epic to improve the test coverage for all the operators supported within Auron. This would greatly help reduce a lot of the correctness issues you mentioned.

Thanks for bringing this up.

[xumingming]

Seems we all think it is necessary, great! I will draft a PR to add correctness test related skeleton(will learn a lot from Apache Gluten. Thanks in advance, Gluten!), and then gradually adding all the tests with separate PRs.