Import of problem zip slow with many submission files

[eldering]

Importing example_problems/hello as a zip takes about 10 seconds, and almost all of that time seems spent importing the submissions.

I think the problem is the double for loop over the files in the zip at

domjudge/webapp/src/Service/ImportProblemService.php

Lines 635 to 659 in 01b754a

	for ($j = 0; $j < $zip->numFiles; $j++) {
	$path = $zip->getNameIndex($j);
	if (!Utils::startsWith($path, 'submissions/')) {
	// Skipping non-submission files silently.
	continue;
	}
	$pathComponents = explode('/', $path);
	if (!((count($pathComponents) == 3 && !empty($pathComponents[2])) ||
	(count($pathComponents) == 4 && empty($pathComponents[3])))) {
	// Skipping files and directories at the wrong level.
	// Note that multi-file submissions sit in a subdirectory.
	continue;
	}
	if (count($pathComponents) == 3) {
	// Single file submission
	$files = [$pathComponents[2]];
	$indices = [$j];
	} else {
	// Multi file submission
	$files = [];
	$indices = [];
	$length = mb_strrpos($path, '/') + 1;
	$prefix = mb_substr($path, 0, $length);
	for ($k = 0; $k < $zip->numFiles; $k++) {

, which may become slow when there's in the case of hello 223 files/directories in the zip file.

[meisterT]

I tried to investigate a bit, the double loop doesn't seem to be an issue. Also we only entering the inner loop for multi-file submissions - for the hello problem we only have a few of these. Commenting that code made it barely faster.

[meisterT]

Elaborating here a little bit more: the slow part is that we create all judging runs and judge tasks (one per test case and judging).

We discussed in Luxor that it we could have a special queuetask to do that instead per judging and let the judgedaemon create the actual judging_run and judge tasks for this submission.

If we do so, also rejudgings should be a lot faster.

[meisterT]

I looked into this again, and it gets a lot faster (before 21s, after 4s) if I disable the event log here:
https://github.com/DOMjudge/domjudge/blob/main/webapp/src/Service/SubmissionService.php#L637-L641

Slow part in there is the internal API request (goes up from 4s to 5s if I do the rest but not the internal API request): https://github.com/DOMjudge/domjudge/blob/main/webapp/src/Service/EventLogService.php#L319

The interesting part seems to be that each of these requests seems to take longer, they take around 5-20ms for the first few requests and can go up to 200ms and more per request:

Now I first thought that this increases with number of submissions in the database, but that doesn't seem to be true. Perhaps there is some other inefficiency in the stacked subrequests that grows with the number of subrequests within one parent request?

[nickygerritsen]

Did you run this in prod mode? In dev mode queries are logged and the more queries you do the slower this gets.

[meisterT]

No, I have used dev mode and also enabled the profiler, otherwise I would be blind. But that's a sensible explanation, I'll try later with prod mode and see how overall timings change.

[nickygerritsen]

You can try $em->getConnection()->getConfiguration()->setSQLLogger(null); to disable query logging at the beginning of the submission service.