fix(scheduler): relax self-referential pod affinity for gang bootstrap

[pfernandes21]

What type of PR is this?

/kind bug

What this PR does / why we need it:

Volcano evaluates inter-pod affinity per task against pods that are already placed, ignoring not-yet-scheduled members of the same gang (podgroup). A gang whose required pod affinity can only be satisfied by its own members therefore deadlocks at the bootstrap step:

• The first member of the gang matches no placed pod on any node (no sibling is scheduled yet).
• InterPodAffinity.Filter returns UnschedulableAndUnresolvable (ErrReasonAffinityRulesNotMatch).
• util.ShouldAbort treats that as a hard failure, so the whole podgroup is wedged and never schedules.

This is exactly the shape of our multi-node EFA training jobs, where every worker carries a required pod affinity that co-locates workers within a shared karpenter.k8s.aws/capacity-reservation-id topology. kube-scheduler has a built-in escape hatch for "the first pod in a self-affine series" (satisfyPodAffinity), but it only fires when no pod anywhere in the cluster matches the selector. When ambient pods match the selector (e.g. other role=worker pods, which is the live behavior today because matchLabelKeys is dropped by the older FlytePropeller), that escape hatch no longer fires and the gang deadlocks.

This PR adds a narrow, gang-aware relaxation in the predicates plugin. When the InterPodAffinity filter fails on the affinity rule (not anti-affinity), we let the bootstrap member through iff:

1. the task declares at least one required pod-affinity term,
2. it belongs to a known gang (podgroup) with more than one member,
3. no member of that gang occupies a node yet (bootstrap phase only), and
4. every required term is satisfiable by a sibling member of the gang.

Because Volcano allocates a gang's members sequentially and updates the node snapshot after each allocation, every subsequent member is still subject to the standard filter and lands in the bootstrap member's topology domain — so the gang stays co-located. Anti-affinity failures are never relaxed (they return plain Unschedulable, not UnschedulableAndUnresolvable, and the reason check excludes them).

Which issue(s) this PR fixes:

Refs volcano-sh#3845

Special notes for your reviewer:

• The relaxation is gated on filterStatus.Reason == interpodaffinity.ErrReasonAffinityRulesNotMatch, so only the pod-affinity rule is ever relaxed; pod anti-affinity and existing-pod anti-affinity are untouched.
• matchLabelKeys (k8s 1.29+) is honored when computing whether a sibling satisfies a term, mirroring how kube-scheduler expands a PodAffinityTerm in PreFilter. This keeps the relaxation correct once matchLabelKeys is properly propagated.
• New test TestGangPodAffinityBootstrap covers two scenarios:
  • isolated-gang-bootstraps: an isolated self-affine gang (also covered by kube-scheduler's own escape hatch) still schedules both members into the shared topology domain.
  • gang-bootstraps-past-ambient-match: an ambient running pod matches the gang's selector in a different topology domain, defeating kube-scheduler's escape hatch. This reproduces the real deadlock; with the patch the gang bootstraps and co-locates in the reserved domain. Verified that neutralizing the relaxation makes only this case fail, confirming the fix is necessary.
• go test ./pkg/scheduler/plugins/predicates/, go vet, and gofmt are all clean; the scheduler binary builds.

Does this PR introduce a user-facing change?

Allow gang (podgroup) scheduling to bootstrap when every member carries a self-referential required pod affinity (e.g. multi-node EFA training jobs co-located within a capacity-reservation topology), which previously deadlocked because InterPodAffinity only evaluates already-placed pods.

Link to Devin session: https://app.devin.ai/sessions/109a77f123df47ef8618364848dba0be
Requested by: @pfernandes21

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment, CI, and merge conflict monitoring