[SPARK-42478] Make a serializable jobTrackerId instead of a non-serializable JobID in FileWriterFactory

### What changes were proposed in this pull request?

Make a serializable jobTrackerId instead of a non-serializable JobID in FileWriterFactory

### Why are the changes needed?

[SPARK-41448](https://issues.apache.org/jira/browse/SPARK-41448) make consistent MR job IDs in FileBatchWriter and FileFormatWriter, but it breaks a serializable issue, JobId is non-serializable.

### Does this PR introduce _any_ user-facing change?

No

### How was this patch tested?

GA

Closes #40064 from Yikf/write-job-id.

Authored-by: Yikf <yikaifei@apache.org>
Signed-off-by: Wenchen Fan <wenchen@databricks.com>

Author: yikf

sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/FileWriterFactory.scala

@@ -30,7 +30,12 @@ case class FileWriterFactory (
     description: WriteJobDescription,
     committer: FileCommitProtocol) extends DataWriterFactory {
 
-  private val jobId = SparkHadoopWriterUtils.createJobID(new Date, 0)
+  // SPARK-42478: jobId across tasks should be consistent to meet the contract
+  // expected by Hadoop committers, but `JobId` cannot be serialized.
+  // thus, persist the serializable jobTrackerID in the class and make jobId a
+  // transient lazy val which recreates it each time to ensure jobId is unique.
+  private[this] val jobTrackerID = SparkHadoopWriterUtils.createJobTrackerID(new Date)
+  @transient private lazy val jobId = SparkHadoopWriterUtils.createJobID(jobTrackerID, 0)
 
   override def createWriter(partitionId: Int, realTaskId: Long): DataWriter[InternalRow] = {
     val taskAttemptContext = createTaskAttemptContext(partitionId)