feat: Drop native_comet as a valid option for COMET_NATIVE_SCAN_IMPL config

[andygrove]

Which issue does this PR close?

Part of #2177

Rationale for this change

This is the next logical step towards removing support for the legacy native_comet Parquet scan implementation.

Note that this PR does not remove the native_comet implementation. It just stops Comet from using it as a scan type.

What changes are included in this PR?

• Drop native_comet as a valid option for COMET_NATIVE_SCAN_IMPL config
• Update CI workflows to stop specifying native_comet
• Ignore tests that only test native_comet. I did not remove them because some of them are tests for v2 scans and we may want to support v2 scans in the future with native_datafusion and native_iceberg_compat scans.

How are these changes tested?

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 58.60%. Comparing base (f09f8af) to head (0dc5ef8).
⚠️ Report is 917 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #3358      +/-   ##
============================================
+ Coverage     56.12%   58.60%   +2.47%     
- Complexity      976     1404     +428     
============================================
  Files           119      175      +56     
  Lines         11743    16162    +4419     
  Branches       2251     2681     +430     
============================================
+ Hits           6591     9471    +2880     
- Misses         4012     5350    +1338     
- Partials       1140     1341     +201     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[parthchandra]

But DSV2 isn't supported by native_iceberg_compat so what will happen there (what if its a Parquet file with complex types)?

[andygrove]

Comet will fall back to Spark for v2 scans until we add the support for v2 scans using native iceberg compat.

[andygrove]

But DSV2 isn't supported by native_iceberg_compat so what will happen there (what if its a Parquet file with complex types)?

The current v2 support using native_comet wouldn't support the complex type case would it?

[mbutrovich]

This makes sense to me. I think a V2 compatible native_datafusion-backed Parquet scan will be pretty straightforward to implement after #3349, so we could close this gap quickly if we needed to. That said, V2 Parquet scans are still opt in so I'd be surprised if too many people noticed the feature gap at the moment.

[andygrove]

This makes sense to me. I think a V2 compatible native_datafusion-backed Parquet scan will be pretty straightforward to implement after #3349, so we could close this gap quickly if we needed to. That said, V2 Parquet scans are still opt in so I'd be surprised if too many people noticed the feature gap at the moment.

Thanks @mbutrovich. @parthchandra does this make sense? Any objection to moving forward with this change?