Proposal: Remove CometFuzz module

[andygrove]

What is the problem the feature request solves?

I would like to propose removeing the CometFuzz standalone testing module, and instead focus on the new SQL file based testing approach introduced in #3328.

Reasons:

• The new approach is catching far more bugs than CometFuzz
• CometFuzz is really just a proof-of-concept and has not been maintained
• CometFuzz does not run in CI
• CometFuzz only covers a small subset of expressions

Describe the potential solution

Standardize on #3328 and leverage agentic coding tools to add additional edge cases for testing as we add support for new expressions

Additional context

No response

[andygrove]

@wForget @mbutrovich @comphead @manuzhang @psvri since you have all contributed to CometFuzz, WDYT? Any objections?

[comphead]

Thanks @andygrove I dont have strong opinion here.

IMO fuzz testing is not the same as integration testing, the current test approach follows the pattern we have already dropped in #2854 in favor of unit tests and fuzz tests. The dropped approach ran spark builtin functions that lives in spark examples registry, but fuzz found more issues back in the day.

I would say both approaches can coexist, with running heavyweight fuzz testing on prerelease checks.
However if unit tests are covered enough(all possible datatypes including edge cases, which pretty much easy to generate with LLMs) for each built in function then it should be enough just having unit test suite

[kazuyukitanimura]

If we decide to delete the fuzz test, it may be a good idea to run it one more time to make sure we do not leave anything suspicious before deleting. If we find issues out the run, we should file issues or add some specific tests

[manuzhang]

I believe agentic coding tools can do better "fuzz" testing than our pre-defined rules.

[Shekharrajak]

Seems like CometFuzz will be redundant and those test cases will already be covered with new approach and must be removed.