MultiResourceItemWriter creates files with varying line count

[spring-projects-issues]

Raja Rajan R S opened BATCH-1868 and commented

When it is required to create output files with MultiResourceItemWriter with all files having the exact line count as mentioned in the itemCountLimitPerResource attribute, Spring batch creates files with varying line counts (varying between the itemCountLimitPerResource value and commit-interval value). The attached class FixedLinesMultiResourceItemWriter.java creates the multiple files with every file having exactly same no of lines as defined by itemCountLimitPerResource attribute.
The usage of the class in the configuration should be as given below.

<beans:bean id="fixedLineCountWriter" class="org.springframework.batch.item.file.FixedLinesMultiResourceItemWriter" scope="step">
   <beans:property name="name" value="fw1" />
   <beans:property name="resource"
         value="file:#{jobParameters['output.file.path.first']}" />
   <beans:property name="resourceSuffixCreator" ref="suffixCreator" />
   <beans:property name="saveState" value="true" />
   <beans:property name="itemCountLimitPerResource" value="250000" />
   <beans:property name="fields" value="comma,seperated,field,names,as,in,the,POJO,class" />
   <beans:property name="delimitter" value="&quot;|&quot;" />
</beans:bean>

where the value for the property fields should be the comma seperate list of field names to be printed in the output file. Field names should always match the name of the attributesas declared in the POJO class definition. Say for example for a Customer Name bean class the value of "fields" property can be "firstName,LastName,MiddleName,title" if these attributes are declared in the bean class.

This has been tested in our spring batch application and confirmed working fine.

---

Affects: 2.1.8

Attachments:

• FixedLinesMultiResourceItemWriter.java (6.83 kB)

[spring-projects-issues]

Raja Rajan R S commented

Note : <beans:property name="delimitter" value=""|"" />
this line should actually be given as <beans:property name="delimitter" value="$ q o u t ;|$ q o u t ;" />

[spring-projects-issues]

Brice Dutheil commented

Actually this does not help with different writers.

I think the real issue is that the check is done after writing, while it should do that before, and eventually create a new resource if needed.

if (!opened) {
     File file = setResourceToDelegate();
     // create only if write is called
     file.createNewFile();
     Assert.state(file.canWrite(), "Output resource " + file.getAbsolutePath() + " must be writable");
     delegate.open(new ExecutionContext());
     opened = true;
}
delegate.write(items);
currentResourceItemCount += items.size();
if (currentResourceItemCount >= itemCountLimitPerResource) {
     delegate.close();
     resourceIndex++;
     currentResourceItemCount = 0;
     setResourceToDelegate();
     opened = false;
}

EDIT : unfortunately there's the transaction issue here, that might cause problems :

Middling with the code a litle bit I came up with these changes, still the transaction fails because the stream is closed, but I don't know yet how to get manage that part.

	public void write(List<? extends T> items) throws Exception {
    openNewResourceIfNecessary();

		currentResourceItemCount += items.size();
    if (currentResourceItemCount >= itemCountLimitPerResource) {
      closeCurrentResource();
      openNewResourceIfNecessary();
    }
    delegate.write(items);
	}


  private void closeCurrentResource() throws IOException {
    delegate.close();
    resourceIndex++;
    currentResourceItemCount = 0;
    setResourceToDelegate();
    opened = false;
  }


  private void openNewResourceIfNecessary() throws IOException {
    if (!opened) {
      File file = setResourceToDelegate();
      // create only if write is called
      file.createNewFile();
      Assert.state(file.canWrite(), "Output resource " + file.getAbsolutePath() + " must be writable");
      delegate.open(new ExecutionContext());
      opened = true;
    }
  }

[fmbenhassine]

Duplicated by #4660

MCVE: #4660 (comment)