Significant difference between scoreboard result and test resuls.

[Moohaa]

Hi, My solution typically takes an average of 2.4 minutes to finish processing 1 billion rows. However, on the scoreboard, it shows a duration of 17 minutes. What could be the reason for this significant difference?

[aboullaite]

Not sure why it is much faster in your machine than ours, but the runtime is different and there are many things that inherently will be different. All submissions runs within the same machine

[samir-]

@sqrt-minus-one i noticed your solution uses 16 threads. the judge machine is an 8vcores one so that must be a first diff.

3 sec solution is not possible unless your file is already cached into memory. 22go is impossible to load from SSD given the raw hardware speed :) so the only explanation is first run takes much much longer then the next runs benefits from the cached file in memory and that can explain the average but the median is still odd .

[sqrt-negativeone]

@samir- Oh you are right I haven't considered the disk caches! after clearing the caches it jumped to 19s which makes sense now 😆

[aboullaite]

@Moohaa your code now runs under 6min :) it reflects what you found

[Moohaa]

Yes, thank you @aboullaite .
The issue was that I used Runtime.getRuntime().availableProcessors(), and it returned 1 in the judge VM. This is a known problem https://repost.aws/questions/QUDkXshWl4RiKuTd_qdDMVxw/jvm-runtime-getruntime-availableprocessors-returns-1-on-aws-ecs-fargate.