concurrency with deblur using `PLUGIN_LAUNCHER = qiita-plugin-launcher`

[sjanssen2]

I seem to constantly running into an concurrency issue. I've set up a local Qiita instance on a Ubuntu virtual machine, based on the example nginx configuration, i.e. with one master and three workers, and installed all plugins necessary for 16S processing.

Creating a test study with just 3 samples let me process up to the timming:

However, deblur fails to finish. It executes the 4 deblur step with additional 4 SEPP steps but to my surprise, it seems to trigger two posts to /qiita_db/jobs/JOBID/complete/ from two different workers:

==> qiita_21177.log <==
[I 240301 15:55:36 web:2161] 200 POST /qiita_db/jobs/00e919d2-b9b6-458d-bc72-3bfb3cb3bb89/complete/ (127.0.0.1) 68.18ms

==> qiita_21175.log <==
[I 240301 15:55:40 web:2161] 200 POST /qiita_db/jobs/00e919d2-b9b6-458d-bc72-3bfb3cb3bb89/complete/ (127.0.0.1) 603.42ms

The admin error table lists Cannot change the step of a job whose status is not 'running', jobid=00e919d2-b9b6-458d-bc72-3bfb3cb3bb89

This might be due to the fact, that deblur generats two artifacts?! Referring to the different processes that are triggered

qiita/scripts/qiita-recover-jobs

Lines 144 to 155 in 13eafdf

	def qiita_recover_jobs():
	# general full processing pipeline, as an example a deblur job as it yields
	# two artifacts, each new line represents a new job, each idented block a
	# waiting job
	# -> deblur
	# -> complete_job -> release_validator
	# -> validate biom 1
	# -> release_validator
	# -> complete_job -> create artifact
	# -> validate biom 2
	# -> release_validator
	# -> complete_job -> create artifact

I see that the jobid 00e919d2-b9b6-458d-bc72-3bfb3cb3bb89 belongs to the validate job of the reference-hit.biom artifact

SELECT * FROM qiita.processing_job WHERE processing_job_id = '00e919d2-b9b6-458d-bc72-3bfb3cb3bb89';:

          processing_job_id           |     email      | command_id |                                                                                                                                                                                                                                                                                                                                                                                                   command_parameters                                                                                                                                                                                                                                                                                                                                                                                                    | processing_job_status_id | logging_id |         heartbeat          |                            step                             | pending | hidden | external_job_id 
--------------------------------------+----------------+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------------+------------+----------------------------+-------------------------------------------------------------+---------+--------+-----------------
 00e919d2-b9b6-458d-bc72-3bfb3cb3bb89 | stefan@foo.bar |         29 | {"analysis": null, "artifact_type": "BIOM", "files": "{\"biom\": [\"/home/qiita/Git/qiita-spots/qiita/qiita_db/support_files/production_data/working_dir/1ada410d-906a-4b44-b823-bfe97fa517fd/deblur_out/deblured/reference-hit.biom\"], \"preprocessed_fasta\": [\"/home/qiita/Git/qiita-spots/qiita/qiita_db/support_files/production_data/working_dir/1ada410d-906a-4b44-b823-bfe97fa517fd/deblur_out/deblured/reference-hit.seqs.fa\"], \"plain_text\": [\"/home/qiita/Git/qiita-spots/qiita/qiita_db/support_files/production_data/working_dir/1ada410d-906a-4b44-b823-bfe97fa517fd/deblur_out/deblured/insertion_tree.relabelled.tre\"]}", "name": "dflt_name", "provenance": "{\"job\": \"1ada410d-906a-4b44-b823-bfe97fa517fd\", \"cmd_out_id\": 13, \"name\": \"deblur reference hit table\"}", "template": 1} |                        6 |            | 2024-03-01 15:55:26.779568 | Completing via c806b5d8-ff9e-4fce-be70-781fcdbfe0ef [33894] |         | f      | 33490

• Where exactly are the POSTs to /qiita_db/jobs/JOBID/complete/ triggered?
• Why twice with the very same job.id?

[sjanssen2]

I can resolve avoid this situation by setting

# Hard upper-limit on concurrently running validator jobs
JOB_SCHEDULER_PROCESSING_QUEUE_COUNT = 1

in qiita's configuration.

[antgonza]

• Where exactly are the POSTs to /qiita_db/jobs/JOBID/complete/ triggered?: This is controlled by the qiita_client, specifically here.
• Why twice with the very same job.id?: This is actually surprising, never seen this and AFAIK this will should break other things as it will create the same job twice - are you sure is the exact same command? Note that you can use trace to figure out all the jobs generated by a single job (like 00e919d2-b9b6-458d-bc72-3bfb3cb3bb89). BTW are you sure this break things?
• JOB_SCHEDULER_PROCESSING_QUEUE_COUNT = 1: in qiita this is set to 2 so that's not it and my guess is that my guess what's happening is that you are submitting the 2 validators twice.

[antgonza]

Closing for now, please reopen if you have further questions.

[sjanssen2]

Hi @antgonza I again stumbled upon this issue. I learned about the "dependent_jobs_list". If JOB_SCHEDULER_PROCESSING_QUEUE_COUNT is set to 1, there are no dependent jobs, and validation runs through smoothly:

qiita/qiita_core/support_files/config_test.cfg

Line 141 in 0a29ac2

JOB_SCHEDULER_PROCESSING_QUEUE_COUNT = 2

However, if you set to > 1, then the list of dependent jobs is not empty anymore and I wonder how dependent jobs are monitored.

qiita/qiita_db/processing_job.py

Lines 1108 to 1114 in 0a29ac2

	p = Process(target=launcher['function'],
	args=(plugin_env_script,
	plugin_start_script,
	url,
	self.id,
	job_dir))
	p.start()

I don't directly see retrieval of the process ID - like here:

qiita/qiita_db/processing_job.py

Lines 1127 to 1128 in 0a29ac2

	if job_id is not None:
	self.external_id = job_id

and wonder how Qiita can poll for job completion?!

[antgonza]

I don't think they are actually monitored but when they are done they "submit" your state while the "lead" keeps running until completion. This happens in in _complete_artifact_transformation, where qiita_config.job_scheduler_dependency_q_cnt is used. Hope this helps.

Anyway, one of the things we have discussed in the past is closer slurm integration and this might be one of those things that could help us push things forward.

For your use case, do you think setting to 1 will "help fixing" the issue?

[sjanssen2]

Setting to 1 fixes it, but I had the feeling this solution is shying away from solving the problem :-)