Description
Jörg Gottschling opened BATCH-1227 and commented
I recently (had to) thought about scalability and performance in one of our largest batch jobs. One problem there is, that our deployment is on system B, this is also were all the business logic and some additional data exists, we read from a database from system A, process them using the business logic on B and write to a database on system C.
The bottleneck is the network. (No, we can not deploy on A or C. :-( )
Up to now it's made with our homegrown Spring based framework, similar to Spring Batch. But will migrate now to Spring Batch for various reasons.
I thought about implementing special delegate Reader, which is working in it's own thread, some what similar to what I have understand how actors work in Erlang or Scala work. My idea is, that the Reader writes into a queue, from which the client (the Step!?) reads. While the Step processes the data, the reader can go own reading, unless the queue reaches a configured limit. After processing the chunk will be handed over to the writer. One could imagine, also to implement it, using a queue. But it think this is not perhaps not needed here and I do not know how it affects transaction handling, etc.
I think to implement a Writer like that would be very hard. Implementing a Processor that way may be impossible, because it has to return the item. So perhaps this is better located at Step level?
What do you think about this? And is this what you refer to as SEDA-Style architecture?
Affects: 2.0.0