Skip to content

Add blocking queue item reader/writer to enable implementing the staged event-driven architecture in batch jobs #2350

Closed
@spring-projects-issues

Description

@spring-projects-issues

Jörg Gottschling opened BATCH-1227 and commented

I recently (had to) thought about scalability and performance in one of our largest batch jobs. One problem there is, that our deployment is on system B, this is also were all the business logic and some additional data exists, we read from a database from system A, process them using the business logic on B and write to a database on system C.
The bottleneck is the network. (No, we can not deploy on A or C. :-( )

Up to now it's made with our homegrown Spring based framework, similar to Spring Batch. But will migrate now to Spring Batch for various reasons.

I thought about implementing special delegate Reader, which is working in it's own thread, some what similar to what I have understand how actors work in Erlang or Scala work. My idea is, that the Reader writes into a queue, from which the client (the Step!?) reads. While the Step processes the data, the reader can go own reading, unless the queue reaches a configured limit. After processing the chunk will be handed over to the writer. One could imagine, also to implement it, using a queue. But it think this is not perhaps not needed here and I do not know how it affects transaction handling, etc.

I think to implement a Writer like that would be very hard. Implementing a Processor that way may be impossible, because it has to return the item. So perhaps this is better located at Step level?

What do you think about this? And is this what you refer to as SEDA-Style architecture?


Affects: 2.0.0

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions