FlatFileItemRead isnt respecting the maxitemcounter [BATCH-2801]

[spring-projects-issues]

Jesús Ponte opened BATCH-2801 and commented

I'm doing a process with Spring Batch multi-thread. My process receives a large txt file (> 100k lines). I want each thread to process X lines of the file and do X process with the information to gain time.

I am using the FlatFileItemRead class passing the initial and final line to each thread. I'm doing tests with 19k lines, 3 and 4 threads and I cannot see why the first thread if it starts and stops in the correct line, but the rest of threads start in the correct line but not end in the correct one, read the file until the end.

The xml configuration and the rangepartitioner files are attached.

My example with 19k lines in file and 3 threads:

Thread: 1
Initial Index: 1
Final Index: 6333

Thread: 2
Index Initial: 6334
Final Index: 12666

Thread: 3
Index Initial: 12667
Final Index: 19000

Thread 1 reads up to 6333. Thread 2 reads up to 19000 instead of reading up to 12666 and thread 3 reads up to 19000.

You can see an example created by Mahmoud Ben Hassine on his GitHub see this

His example (15 lines in file and 3 threads):

• SimpleAsyncTaskExecutor-1 reads from 1 to 5 as expected
• SimpleAsyncTaskExecutor-2 reads from 6 to 15 which is incorrect (it should read from 5 to 10 only)
• SimpleAsyncTaskExecutor-3 reads from 11 to 15 as expected

 

 

---

Affects: 2.1.9, 4.1.1

Reference URL: https://stackoverflow.com/questions/55227133/why-do-these-threads-read-the-entire-file-instead-of-maxitemcounter

Attachments:

• RangePartitioner.java (1.71 kB)
• XML_Configuration.xml (5.10 kB)

[hpoettker]

I think this issue can be closed as the issue is actually a user error. The value of maxItemCount is increased for each partition although it should stay constant.

I've pushed a sample with a fixed Partitioner here.