PBSProProvider doesn't detect jobs completing

[WardLT]

Describe the bug
PBSProProvider fails to detect a job finishing, which means blocks stay active and Parsl will not start new ones.

The error in the logs is:

1752246824.600159 2025-07-11 15:13:44 MainProcess-931080 JobStatusPoller-Timer-Thread-140649297520800-140648986441472 parsl.providers.pbspro.pbspro:103 _status WARNING: qstat failed with retcode:35 STDOUT:{
    "timestamp":1752246824,
    "pbs_version":"2024.1.2.20241017100211",
    "pbs_server":"polaris-pbs-01.hsn.cm.polaris.alcf.anl.gov"
} STDERR:qstat: 5487654.polaris-pbs-01.hsn.cm.polaris.alcf.anl.gov Job has finished, use -x or -H to obtain historical job information

To Reproduce
Run Parsl with a short walltime for a job.

Expected behavior
New blocks are submitted after old

Environment

• Python version: 3.12
• Parsl version: 2025.07.07

Distributed Environment
ALCF's Polaris. DFK resides on the head node

[WardLT]

@cms21 , have you been seeing this too?

[cms21]

I have not seen this @WardLT . Do you know what flags parsl uses with qstat? If you add -x as the error suggests you can retrieve the record for this job.

Looking at this job, your log example is trying to query at 2025-07-11 15:13:44. Do you know the timezone of your log timestamps? What was the timestamp of the first failure in your logs? The mtime (which is the last time the job was modified) in the pbs record is mtime = Fri Jul 11 10:13:30 2025 in Chicago time.

If your log timestamp is in UTC then it couldn't find the job 14 seconds after it was last modified (e.g. marked as finished). I'm not sure how long a job record should be available without the -x flag, I don't think it is long. I also don't think this has changed recently. I'm not sure why this is happening now and not before.

If you always include -x on the qstat line, you shouldn't run into this problem