[SPARK-49300][CORE][3.5] Fix Hadoop delegation token leak when tokenRenewalInterval is not set (apache#552)

Backport from master.

### What changes were proposed in this pull request?

Cancel delegation token once they are used in `getTokenRenewalInterval`

### Why are the changes needed?

When `tokenRenewalInterval` is not set, HadoopFSDelegationTokenProvider#getTokenRenewalInterval will fetch some tokens and renew them to get a interval value.
https://github.com/apache/spark/blob/dd259b0b27841e6dd7c07f8ca3cc05d275863dd5/core/src/main/scala/org/apache/spark/deploy/security/HadoopFSDelegationTokenProvider.scala#L60-L64
These tokens do not call cancel(), resulting in a large number of existing tokens on HDFS not being cleared in a timely manner, causing additional pressure on the HDFS server.
### Does this PR introduce _any_ user-facing change?

no

### How was this patch tested?

manual test

### Was this patch authored or co-authored using generative AI tooling?

no

Closes apache#47823 from zhangshuyan0/branch-3.5-zsy.

Authored-by: zhangshuyan <zhangshuyan@apache.org>

Signed-off-by: Chao Sun <chao@openai.com>
Co-authored-by: zhangshuyan <zhangshuyan@apache.org>

Author: 2 people

core/src/main/scala/org/apache/spark/deploy/security/HadoopFSDelegationTokenProvider.scala

@@ -146,6 +146,9 @@ private[deploy] class HadoopFSDelegationTokenProvider
         val tokenKind = token.getKind.toString
         val interval = newExpiration - getIssueDate(tokenKind, identifier)
         logInfo(s"Renewal interval is $interval for token $tokenKind")
+        // The token here is only used to obtain renewal intervals. We should cancel it in
+        // a timely manner to avoid causing additional pressure on the server.
+        token.cancel(hadoopConf)
         interval
       }.toOption
     }