Add Spark Connect serving to hudi-lakehouse with ADBC client integrations #20234

Summary

Provide an optional Helm-deployable Spark Connect endpoint for Hudi queries and writes, with ADBC access from hudi-agent-gateway and BI tools such as Superset.

This issue tracks the implementation discussed in #19843. Local testing demonstrated Hudi reads and writes through Foundry ADBC with BI tools such as Superset; the next step is to package the endpoint and integrations and validate them through Helm.

Scope

• Provide a Hudi-enabled Spark Connect image and Helm chart, reusing the lakehouse catalog and object storage.
• Integrate hudi-agent-gateway through ADBC while preserving its existing tool interfaces and SELECT-only guardrails.
• Provide integration examples for BI tools such as Superset, supporting direct queries and writes.
• Add repeatable end-to-end validation and a complete quickstart.

Direct-client write coverage includes table creation, INSERT, UPDATE, DELETE, and MERGE INTO. Agent gateway write tools are out of scope.

Sub-issues

• Add a Helm-deployable Spark Connect endpoint #20231 — Add a Helm-deployable Spark Connect endpoint
• Integrate hudi-agent-gateway and BI clients with Spark Connect #20232 — Integrate hudi-agent-gateway and BI clients with Spark Connect
• Add end-to-end validation and documentation for Spark Connect #20233 — Add end-to-end validation and documentation for Spark Connect

Completion Criteria

• A clean local Kubernetes environment can deploy the endpoint using the quickstart.
• Agent/MCP queries work through Gateway → ADBC → Spark Connect → Hudi, with read-only guardrails enforced.
• The selected BI tools, such as Superset, can query and write Hudi data, with result readback and rendering.
• Tests cover the scoped SQL operations, external-commit visibility, and persistence across endpoint restarts.
• Documentation records tested versions, setup, cleanup, and known limitations.

Follow-up Work

Broader JDBC compatibility, gateway write tools, and production-scale security, concurrency, and isolation validation are separate follow-up work.