Retry support for exceptions in transaction commit phase without re-executing reader

[daanvdh]

Expected Behavior

By adding Throwables to a (not yet existing) configuration, the batch framework will catch and swallow these exceptions if they where fired during the commit phase of the transaction and add the ChunkContext including the attributeQueue so that this chunk can be retried without having to execute the reader.

Current Behavior

Currently there is no retry mechanism in place where ChunkContext has status completed (correctly) and we only want to retry the processor and writer.

Work around implementation details

Implementing this feature currently requires the copying and pasting from some classes, with only minor changes to the code. This will be difficult to maintain if there are framework changes.

• copy paste ChunkOrientedTasklet and remove line "chunkContext.removeAttribute(INPUTS_KEY);"
• set tour own version of ChunkOrientedTasklet by using TaskletStep::setTasklet in the jobConfiguration.
• copy paste StepContextRepeatCallback and add a catch statement catching exceptions from the transaction commit. In the catch store the ChunkContext in the attributeQueue if the exception if swallowed. Also set the completed status to false.
• Extend TaskletStep to override doExecute, only to use our own version of StepContextRepeatCallback.
• Use our version of TaskletStep in the jobConfiguration.

Context

Our use case is that we read multiple xml files concurrently that contain data to either create or update entities in a db. The system must be able to handle multiple xml files that possibly contain updates to the same entities. The current design is that a batch job reads in a single xml file, the reader reads a chunk of xml data and creates a pojo per entity to create/update. These pojos are input to the processor which fetches (if present) the existing entity and updates/creates it. Then the writer writes them to the db.

Currently if 2 xml files that contain an update to the same entity are executed concurrently in 2 different batch jobs, an OptimisticLockingException can occur in the commit phase. That is because both processors fetched the existing entity from the db before a writer wrote its changes. Simply re-executing the processor and writer on the same pojo will fix this.

[fmbenhassine]

Thank you for opening this issue.

Current Behavior
Currently there is no retry mechanism in place where ChunkContext has status completed (correctly) and we only want to retry the processor and writer.

I am not sure I understand the problem with this. If the ChunkContext is marked as completed correctly, why would one want or need to retry the chunk?

For this kind of non trivial issues, and in order to address this ticket in an efficient way, we are expecting you to provide a minimal example that we can inspect to understand the problem you are describing and what you are requesting as a new feature. Thank you for your collaboration.

BTW, we are going to revisit the current implementation of the chunk-oriented processing model (see #3950), which might impact the way we introduce new features related to fault-tolerance and concurrency.

[rowland66]

Hi @fmbenhassine, I am struggling with the same problem. It looks to me like ChunkOrientedTasklet.execute() sets the chunkContext to complete after it processes the chunk. However ChunkOrientedTasklet.execute() is called from TaskletStep.ChunkTransactionCallback.doInTransaction() which is called from the execution of a TransactionTemplate. The TransactionTemplete commits the transaction, so if an exception is thrown during commit, the chunkContext has already been marked complete. A retry is initiated, but since the chunkContext is complete, the buffered items from the Reader are discarded. This results in the reader reading the next chunk of items, and the retry does not reprocess the chunk of items that failed to commit. Note that when using an ORM like Hibernate, quite a bit of processing takes place during the commit, and even without Hibernate, a failure to commit should not cause a chunk to be dropped.

I think the easiest solution would be for TaskletStep.ChunkTransactionCallback.afterCompletetion() to set ChunkContext complete to false if the transaction status indicates rollback. This would allow the buffered items to be retained, and reprocessed.

[fmbenhassine]

@rowland66 Thank you for sharing your feedback. This is a known old issue (#1826), and the current implementation does not handle exceptions during commits correctly. I attempted to fix the issue with the current implementation (#591), but I discovered that the problem is deeper than that. So I decided to discard the fix and change the implementation altogether for better maintainability in the long term. This is coming in v6 (#3950).