feat(sparkconnect): add service account fallback for Spark Connect server pod

[vikas-saxena02]

Purpose of this PR

Part of #3049. Adds a service account fallback for the Spark Connect server pod,
completing the issue alongside #3092 .

Two changes:

1. A new optional .spec.server.serviceAccount field on the SparkConnect CRD,
   for parity with .spec.driver.serviceAccount on SparkApplication.
2. The server pod now falls back to the --default-service-account controller
   flag introduced in feat(controller): add default service account fallback for SparkApplication driver #3092 when neither the SparkConnect nor its
   server pod template specifies one.

Precedence

Resolved in resolveServerServiceAccount, highest first:

1. .spec.server.serviceAccount
2. .spec.server.template.spec.serviceAccountName
3. --default-service-account

If none is set the field is left empty and the namespace default applies, so
behaviour is unchanged for existing users.

I proposed this ordering on the issue and nobody objected, but it wasn't
explicitly confirmed — happy to change it if the pod template should win over
the dedicated field.

Scope

Server pod only. Per @tariq-hasan's guidance on #3049, the fallback exists
because the driver needs permission to create executor pods; executors don't
have the same requirement. That's also why the field sits on ServerSpec
rather than the embedded SparkPodSpec, which ExecutorSpec shares.

Implementation note

The resolution happens in mutateServerPod after the pod template is copied,
since pod.Spec = template.Spec is a wholesale assignment that would otherwise
discard it. It's also inside the pod.CreationTimestamp.IsZero() guard, as
serviceAccountName is immutable on an existing pod.

Stacked on #3092

This branches off the --default-service-account PR and won't build without it,
so the diff includes that commit. I'll rebase onto master once it merges.

Testing

• resolveServerServiceAccount unit tests covering all four precedence cases
  plus an empty-string server value.
• mutateServerPod specs asserting the resolved value survives the pod template
  copy.
• make generate, make update-crd, make detect-crds-drift — CRD, deepcopy,
  openapi, swagger and Python model regenerated and in sync.
• make go-vet, make build-operator, make unit-test, make helm-unittest
  all pass.

Closes #3049

[vikas-saxena02]

/retest

[vikas-saxena02]

/retest

[vikas-saxena02]

/retest

[vikas-saxena02]

@tariq-hasan Thanks again for the guidance on #3049 — this is the SparkConnect half, stacked on #3092 .

One decision I'd like your call on before this goes further. With the new .spec.server.serviceAccount field there are now three possible sources for the server pod's service account:

1. .spec.server.serviceAccount
2. .spec.server.template.spec.serviceAccountName
3. --default-service-account

I've implemented that ordering, so the dedicated field wins over the pod template. The reasoning is that an explicit API field is a stronger signal of intent than a value buried in a pod template, and it matches how .spec.driver.serviceAccount behaves for SparkApplication. Happy to flip it if you'd rather the template take priority.

Two implementation notes that aren't obvious from the diff:

• The resolution happens in mutateServerPod after the pod template is copied. pod.Spec = template.Spec is a wholesale assignment, so resolving earlier would silently discard the result — two of the specs exist specifically to catch that regression.
• The field is on ServerSpec rather than the embedded SparkPodSpec, since ExecutorSpec shares that struct and would otherwise inherit it. That keeps the scope to the server pod, per your point about the driver needing pod-create permission.

[vikas-saxena02]

/unhold

[tariq-hasan]

Hi @vikas-saxena02! I have added a few comments.

[tariq-hasan]

/assign

[vikas-saxena02]

@tariq-hasan review comments addressed

[tariq-hasan]

Just have a few minor comments. Otherwise lgtm. Thanks.

[tariq-hasan]

/retest

[vikas-saxena02]

/retest

[vikas-saxena02]

@tariq-hasan all tests have passed and the review comments have been addressed

[tariq-hasan]

Thanks!
/lgtm

[vikas-saxena02]

@andreyvelich @vara-bonthu @yuchaoran2011 given @nabuskey is away can yiu guys p;ease approve this as its already marked lgtm by @tariq-hasan

[andreyvelich]

Sure, thanks @vikas-saxena02!
/lgtm
/approve
/cc @RobuRishabh @ChenYi015

[google-oss-prow]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: andreyvelich

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [andreyvelich]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment