EvalStorage createIterator() sometimes crashes with ArrayIndexOutOfBounce Exception

[schadomi7]

causes non-dockerized benchmark test to freeze, dockerized not tested.

does happen more frequently when computing stuff or generating a lot of
messages, but does still happen with low load as well.

ret.add(new ResultPairImpl(expectedResponses.get(i), actualResponses.get(i)));
appears to be responsible

example log: log.txt
more logs: https://git.cs.upb.de/schadomi/hobbit-java-sdk-example/pipelines/23989

@MichaelRoeder told me i should mention him

[MichaelRoeder]

The EvalStorage class seems to contain two problems.

1. The two lists that are used are not thread-safe. That can lead to the loss of responses and if actualResponses.size() < expectedResponses.size(), the Exception is triggered.
2. It assumes that tasks are received in a certain order. All received responses are simply added to the end of the lists. During the evaluation phase, the storage iterates over the results and assumes that responses at the same positions belong together. So it completely ignores the taskIds but relies on the order. However, as soon as a benchmark generates tasks in parallel, this leads to problems.

I would suggest to use the InMemoryEvaluationStorage of hobbit-core version 1.0.10. It is thread-safe and makes sure that the expected and actual responses are mapped to each other based on their task ids. @schadomi7 and myself tested it and it is working fine for their use case.

[smirnp]

@MichaelRoeder , thank you for response.
The basic storage is included for "learning purposes": to track the execution order by hitting the breakpoints and demonstrate an ability to extend it with some additional logic.
I will improve it and will add a comment about InMemoryStorage to the code.

[MichaelRoeder]

The super(1) may solve the first problem but it can not solve problem 2.
No, the DefaultEvaluationStorage that is typically used (not to be confused with the InMemory solution) has to be thread-safe. Otherwise, we can not work with big data 😉

I understand your point and I agree that an easy storage is nice to understand the reason why it is there. However, I would still argue that the InMemoryStorage is not really complicated. Apart from that, it shows the reason why taskIds are used which is something that can not be understood from the current evaluation storage implementation.

However, it is up to you to decide how to proceed 😉
I just wanted to make you aware of the classes that are already existing.

[smirnp]

Fixed in v1.0.1