Skip to content

Conversation

@pm-blanco
Copy link
Collaborator

The last periodic samples job failed with the following error:

Traceback (most recent call last):
  File "/home/runner/work/pyMBE/pyMBE/samples/Beyer2024/globular_protein.py", line 127, in <module>
    os.makedirs('./frames')
  File "<frozen os>", line 225, in makedirs
FileExistsError: [Errno 17] File exists: './frames'
E
======================================================================
ERROR: test_globular_protein (__main__.Test.test_globular_protein)
----------------------------------------------------------------------
multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/software/Python/3.11.3-GCCcore-12.3.0/lib/python3.11/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
                    ^^^^^^^^^^^^^^^^^^^
  File "/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/software/Python/3.11.3-GCCcore-12.3.0/lib/python3.11/multiprocessing/pool.py", line 48, in mapstar
    return list(map(*args))
           ^^^^^^^^^^^^^^^^
  File "/home/runner/work/pyMBE/pyMBE/testsuite/globular_protein_tests.py", line 52, in kernel
    subprocess.check_output(run_command)
  File "/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/software/Python/3.11.3-GCCcore-12.3.0/lib/python3.11/subprocess.py", line 466, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/software/Python/3.11.3-GCCcore-12.3.0/lib/python3.11/subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/home/runner/work/pyMBE/pyMBE/venv/bin/python3.11', PosixPath('/home/runner/work/pyMBE/pyMBE/samples/Beyer2024/globular_protein.py'), '--pdb', '1f6s', '--pH', '2', '--path_to_cg', 'parameters/globular_proteins/1f6s.vtf', '--mode', 'test', '--no_verbose', '--output', '/tmp/tmpbq6xb_mf']' returned non-zero exit status 1.
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/runner/work/pyMBE/pyMBE/testsuite/globular_protein_tests.py", line 63, in test_globular_protein
    results = dict(pool.map(kernel, tasks, chunksize=1))
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/software/Python/3.11.3-GCCcore-12.3.0/lib/python3.11/multiprocessing/pool.py", line 367, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/software/Python/3.11.3-GCCcore-12.3.0/lib/python3.11/multiprocessing/pool.py", line 774, in get
    raise self._value
subprocess.CalledProcessError: Command '['/home/runner/work/pyMBE/pyMBE/venv/bin/python3.11', PosixPath('/home/runner/work/pyMBE/pyMBE/samples/Beyer2024/globular_protein.py'), '--pdb', '1f6s', '--pH', '2', '--path_to_cg', 'parameters/globular_proteins/1f6s.vtf', '--mode', 'test', '--no_verbose', '--output', '/tmp/tmpbq6xb_mf']' returned non-zero exit status 1.

The following tests FAILED:
	  1 - globular_protein_tests (Failed)

When I re-ran the samples job manually on Github it passed and all tests passed locally on my desk computer as well. I suspect that the issue was in the following lines:

if not os.path.exists('./frames'):
    os.makedirs('./frames')

which can result on an error when executing a script with multiple threats if it happens that two (or more) processes execute the first line before the first threat creates a folder in the following one. To avoid this issue I have decided to switch to pathlib instead:

Path("./frames").mkdir(parents=True,
                       exist_ok=True)

I believe that this should solve the issue, although I was not able to reproduce the initial error. I took the chance also to update our changelog.

@pm-blanco pm-blanco added bug Something isn't working ci-improvement labels Sep 5, 2024
@pm-blanco pm-blanco requested a review from jngrad September 5, 2024 09:49
@pm-blanco pm-blanco self-assigned this Sep 5, 2024
@pm-blanco pm-blanco merged commit ec31779 into pyMBE-dev:main Sep 5, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working ci-improvement

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants