The `environment` kwarg in `JobSpec` gets the wrong number of quotation marks if there are spaces

[Andrew-S-Rosen]

The following behavior was observed on v0.9.0 and 79309dc.

When the environment kwarg of JobSpec has spaces in the value of the key-value pair, the resulting environment variable gets the wrong number of quotation marks.
 

from psij import Job, JobAttributes, JobExecutor, JobSpec, ResourceSpecV1

job_executor = JobExecutor.get_instance("slurm")
job = Job(
    JobSpec(
        name="test",
        environment={"QUACC_VASP_PARALLEL_CMD": "srun -N 1 --ntasks-per-node=48 --cpu_bind=cores"},
        executable="/bin/date",
        resources=ResourceSpecV1(node_count = 1),
        attributes=JobAttributes(project_name = 'matgen', custom_attributes = {'slurm.constraint': 'cpu', 'slurm.qos': 'debug'}),
        launcher="single",
    )
)

job_executor.submit(job)

The above example produces the following Slurm script:

#!/bin/bash



#SBATCH --export=ALL

#SBATCH --exclusive

#SBATCH --nodes=1





#SBATCH --time=0:10:00


#SBATCH --account="matgen"


#SBATCH --constraint="cpu"
#SBATCH --qos="debug"


#SBATCH -e /dev/null
#SBATCH -o /dev/null

#SBATCH --export=QUACC_VASP_PARALLEL_CMD=''"'"'srun -N 1 --ntasks-per-node=48 --cpu_bind=cores'"'"''

exec &>> "/global/homes/r/rosen/.psij/work/slurm/$SLURM_JOB_ID.out"

/bin/bash /global/homes/r/rosen/.psij/work/f65b111560da52a6d41f52a732be0faf4914eb6d7af2230a8de326017979e684/single_launch.sh a0ac935e-5f34-42dc-97a7-703674b23624 '' '' '' /dev/null /dev/null /dev/null /bin/date

echo "$?" > "/global/homes/r/rosen/.psij/work/slurm/$SLURM_JOB_ID.ec"

Note the #SBATCH --export line. It's not just cumbersome --- it actually does change the meaning of the string because it goes from being

"srun -N 1 --ntasks-per-node=48 --cpu_bind=cores"

to

"'srun -N 1 --ntasks-per-node=48 --cpu_bind=cores'"

Doing string operations on this variable can then lead to some unexpected behavior.

This bug is not observed if you just do environment={"hello": "world"}.

[hategan]

Thank you for spotting this.

Strings are escaped using shlex.quote(). The opening and closing single quotes should be there, since they introduce a lexical string. But for some reason, inside those, there are also some escaped single quotes whose origin escapes me. I'll do some digging.

[Andrew-S-Rosen]

Thanks! String handling is always surprisingly more complex and frustrating than one might anticipate...

[hategan]

Thanks! String handling is always surprisingly more complex and frustrating than one might anticipate...

There is much truth in that statement.

Ok, so it looks like environment values are escaped twice. The template renderer does it once and then the piece of code that aggregates the environment variables also does it. The latter isn't necessary. I should have a patch shortly.

[Andrew-S-Rosen]

Many thanks for the speedy bug fix!