--ci-generate pipeline fails when using a test fixture

[yandthj]

We have the following test:

import os

import reframe as rfm
import reframe.utility as util
import reframe.utility.sanity as sn

ValidateResources = util.import_from_module('..utils', 'ValidateResources')

@rfm.simple_test
class build_ior(rfm.CompileOnlyRegressionTest):
    descr = 'Compile IOR'
    valid_systems = ['*']
    valid_prog_environs = ['*']
    build_system = "Make"
    ior_dir = variable(str)

    @run_before('compile')
    def prepare_build(self):
        self.ior_dir = 'ior-3.3.0'
        self.prebuild_cmds += [
            'wget https://github.com/hpc/ior/releases/download/3.3.0/ior-3.3.0.tar.gz',
            f'tar -xzf {self.ior_dir}.tar.gz',
            f'cd {self.ior_dir}',
            f'./configure CC={self.current_environ.cc} MPICC={self.current_environ.cc}',
        ]
    @sanity_function
    def validate_build(self):
        # If compilation fails, the test would fail in any case, so nothing to
        # further validate here.
        return True


@rfm.simple_test
class IOR_cpu(rfm.RunOnlyRegressionTest, ValidateResources):
    descr = 'IOR benchmark'
    valid_systems = ['*']
    valid_prog_environs = ['*']
    executable_opts = ['-v -a POSIX -g -w -r -e -C -F -b 4000m -t 1m -s 1']
    n_nodes = variable(int, value=1)
    mem = 0
    filesystem = parameter(['scratch', 'projects'])
    time_limit = '0d0h15m0s'
    ior_build = fixture(build_ior, scope='environment')

    @run_before('compile')
    def set_executable(self):
        self.executable = os.path.join(
            self.ior_build.stagedir,
            self.ior_build.ior_dir,
            'src', 'ior'
        )

    @run_before('run')
    def set_run_dir(self):
        username = os.getlogin()
        if self.filesystem == 'scratch' or self.filesystem == 'home':
            run_dir = f'/{self.filesystem}/{username}'
        else:
            raise ValueError
        self.prerun_cmds += [f'[ ! -d {run_dir} ] && mkdir {run_dir}']
        # If the run directory wasn't created, the mkdir and cd commands fail,
        # but ReFrame would proceed to run the test in the staging directory,
        # which is not what we want.  Instead, we use "exit" to signal an error.
        self.prerun_cmds += [f'[ ! -d {run_dir} ] && exit 1', f'cd {run_dir}', 
                             'mkdir IOR_$SLURM_JOB_ID', 'cd IOR_$SLURM_JOB_ID']
        self.postrun_cmds += ['cd ../', 'rm -r IOR_$SLURM_JOB_ID']

    @sanity_function
    def assert_solution(self):
        return sn.assert_found(r'Finished\s+:.*', self.stdout)

    @performance_function('MiB/s', perf_key='Mean Read')
    def extract_mean_read(self):
        data = sn.extractall(r'read(.*)', self.stdout, 1, str)[-1]
        data_array = str(data).split()
        return float(data_array[2])

    @performance_function('MiB/s', perf_key='Mean Write')
    def extract_mean_write(self):
        data = sn.extractall(r'write(.*)', self.stdout, 1, str)[-1]
        data_array = str(data).split()
        return float(data_array[2]) 

Using --ci-generate=ior-pipeline/pipeline.yml generates the following pipeline:

default:
cache:
  key: ${CI_COMMIT_REF_SLUG}
  paths:
  - rfm-stage/${CI_COMMIT_SHORT_SHA}
stages:
- rfm-stage-0
- rfm-stage-1
build_ior_f18197df:
  stage: rfm-stage-0
  script:
  - reframe --prefix=rfm-stage/${CI_COMMIT_SHORT_SHA} -C system-settings/settings-swift.py -c tests/ior/ior-source.py  --report-file=build_ior_f18197df-report.json  --report-junit=build_ior_f18197df-report.xml  -n /ebc719b6 -r
  artifacts:
    paths:
    - build_ior_f18197df-report.json
  needs: []
IOR_cpu_1:
  stage: rfm-stage-1
  script:
  - reframe --prefix=rfm-stage/${CI_COMMIT_SHORT_SHA} -C system-settings/settings-swift.py -c tests/ior/ior-source.py  --report-file=IOR_cpu_1-report.json --restore-session=build_ior_f18197df-report.json --report-junit=IOR_cpu_1-report.xml  -n /eed258d9 -r
  artifacts:
    paths:
    - IOR_cpu_1-report.json
  needs:
  - build_ior_f18197df
IOR_cpu_0:
  stage: rfm-stage-1
  script:
  - reframe --prefix=rfm-stage/${CI_COMMIT_SHORT_SHA} -C system-settings/settings-swift.py -c tests/ior/ior-source.py  --report-file=IOR_cpu_0-report.json --restore-session=build_ior_f18197df-report.json --report-junit=IOR_cpu_0-report.xml  -n /e057fe9c -r
  artifacts:
    paths:
    - IOR_cpu_0-report.json
  needs:
  - build_ior_f18197df
build_ior:
  stage: rfm-stage-0
  script:
  - reframe --prefix=rfm-stage/${CI_COMMIT_SHORT_SHA} -C system-settings/settings-swift.py -c tests/ior/ior-source.py  --report-file=build_ior-report.json  --report-junit=build_ior-report.xml  -n /c20f7057 -r
  artifacts:
    paths:
    - build_ior-report.json
  needs: []

Jobs IOR_cpu_0 and IOR_cpu_1 fail with this error:
ERROR: run session stopped: reframe error: could not restore testcase ('build_ior_f18197df', 'swift:standard', 'intelmpi'): not found in the report files

No tests are run in the build_ior_f18197df job so the build_ior_f18197df-report.json is empty. We tried with and without the decorator on build_ior.

[vkarak]

Hi and apologies for the late reply. I cannot reproduce it at the moment since the utils module is missing:

ValidateResources = util.import_from_module('..utils', 'ValidateResources')

Would you mind providing a stand-alone reproducer?

[pranav-sivaraman]

I just ran into this issue. Here is a reproducer based on the stream tutorial examples.

import os
import reframe as rfm
import reframe.utility.sanity as sn
import reframe.core.builtins as rfm_builtins


class build_stream(rfm.CompileOnlyRegressionTest):
    build_system = "SingleSource"
    sourcepath = "stream.c"
    executable = "stream.x"
    array_size = rfm_builtins.parameter([1, 2])

    @rfm_builtins.run_before("compile")
    def prepare_build(self):
        self.build_system.cflags = ["-O3", "-fopenmp"]
        if self.array_size:
            self.build_system.cppflags = [f"-DARRAY_SIZE={self.array_size}"]


@rfm.simple_test
class stream_test(rfm.RunOnlyRegressionTest):
    valid_systems = ["*"]
    valid_prog_environs = ["*"]
    stream_binary = rfm_builtins.fixture(build_stream, scope="environment")
    num_threads = rfm_builtins.parameter([1])
    thread_placement = rfm_builtins.parameter(["true", "close", "spread"])

    @rfm_builtins.run_after("setup")
    def set_executable(self):
        self.executable = os.path.join(self.stream_binary.stagedir, "stream.x")

    @rfm_builtins.run_before("run")
    def setup_threading(self):
        self.env_vars["OMP_NUM_THREADS"] = self.num_threads
        self.env_vars["OMP_PROC_BIND"] = self.thread_placement

    @rfm_builtins.sanity_function
    def validate(self):
        return sn.assert_found(r"Solution Validates", self.stdout)

    @rfm_builtins.performance_function("MB/s")
    def copy_bw(self):
        return sn.extractsingle(r"Copy:\s+(\S+)", self.stdout, 1, float)

    @rfm_builtins.performance_function("MB/s")
    def triad_bw(self):
        return sn.extractsingle(r"Triad:\s+(\S+)", self.stdout, 1, float)

Listing all the tests out normally

[ReFrame Setup]
  version:           4.8.4
  command:           '/Users/psivaram/.local/bin/reframe -c stream.py -l'
  launched by:       psivaram@HPE-C75C4DFH4W
  working directory: '/Users/psivaram/Documents/reframe-repro'
  settings files:    '<builtin>'
  selected system:   'generic'
  check search path: '/Users/psivaram/Documents/reframe-repro/stream.py'
  stage directory:   '/Users/psivaram/Documents/reframe-repro/stage'
  output directory:  '/Users/psivaram/Documents/reframe-repro/output'
  log files:         '/var/folders/kj/9000v5qx40b428qbb160k6sm0000gp/T/rfm-5b6dzl55.log'
  results database:  [off] '/Users/psivaram/.reframe/reports/results.db'

[List of matched checks]
- stream_test %num_threads=1 %thread_placement=true %stream_binary.array_size=1 /836a6bfd
    ^build_stream %array_size=1 ~generic:default+builtin 'stream_binary /6339d247
- stream_test %num_threads=1 %thread_placement=close %stream_binary.array_size=1 /39c9f06c
    ^build_stream %array_size=1 ~generic:default+builtin 'stream_binary /6339d247
- stream_test %num_threads=1 %thread_placement=spread %stream_binary.array_size=1 /39a0c84b
    ^build_stream %array_size=1 ~generic:default+builtin 'stream_binary /6339d247
- stream_test %num_threads=1 %thread_placement=true %stream_binary.array_size=2 /c9055152
    ^build_stream %array_size=2 ~generic:default+builtin 'stream_binary /1c4226c5
- stream_test %num_threads=1 %thread_placement=close %stream_binary.array_size=2 /f91aa8b4
    ^build_stream %array_size=2 ~generic:default+builtin 'stream_binary /1c4226c5
- stream_test %num_threads=1 %thread_placement=spread %stream_binary.array_size=2 /a3797917
    ^build_stream %array_size=2 ~generic:default+builtin 'stream_binary /1c4226c5
Found 6 check(s)

Log file(s) saved in '/var/folders/kj/9000v5qx40b428qbb160k6sm0000gp/T/rfm-5b6dzl55.log'

Specifying hashcode for a fixture

[ReFrame Setup]
  version:           4.8.4
  command:           '/Users/psivaram/.local/bin/reframe -c stream.py -l -n /1c4226c5'
  launched by:       psivaram@HPE-C75C4DFH4W
  working directory: '/Users/psivaram/Documents/reframe-repro'
  settings files:    '<builtin>'
  selected system:   'generic'
  check search path: '/Users/psivaram/Documents/reframe-repro/stream.py'
  stage directory:   '/Users/psivaram/Documents/reframe-repro/stage'
  output directory:  '/Users/psivaram/Documents/reframe-repro/output'
  log files:         '/var/folders/kj/9000v5qx40b428qbb160k6sm0000gp/T/rfm-49oee7h_.log'
  results database:  [off] '/Users/psivaram/.reframe/reports/results.db'

[List of matched checks]
Found 0 check(s)

Log file(s) saved in '/var/folders/kj/9000v5qx40b428qbb160k6sm0000gp/T/rfm-49oee7h_.log'

Specific hashcode for an actual test

[ReFrame Setup]
  version:           4.8.4
  command:           '/Users/psivaram/.local/bin/reframe -c stream.py -l -n /836a6bfd'
  launched by:       psivaram@HPE-C75C4DFH4W
  working directory: '/Users/psivaram/Documents/reframe-repro'
  settings files:    '<builtin>'
  selected system:   'generic'
  check search path: '/Users/psivaram/Documents/reframe-repro/stream.py'
  stage directory:   '/Users/psivaram/Documents/reframe-repro/stage'
  output directory:  '/Users/psivaram/Documents/reframe-repro/output'
  log files:         '/var/folders/kj/9000v5qx40b428qbb160k6sm0000gp/T/rfm-vtwe9f_6.log'
  results database:  [off] '/Users/psivaram/.reframe/reports/results.db'

[List of matched checks]
- stream_test %num_threads=1 %thread_placement=true %stream_binary.array_size=1 /836a6bfd
    ^build_stream %array_size=1 ~generic:default+builtin 'stream_binary /6339d247
Found 1 check(s)

Log file(s) saved in '/var/folders/kj/9000v5qx40b428qbb160k6sm0000gp/T/rfm-vtwe9f_6.log'

Seems like something goes wrong when specifying the hashcodes for fixtures. This is a problem since for the gitlab pipeline it requires specifying the hashcode for each individual step.

[pranav-sivaraman]

Commenting out the following line seems to fix the issue. But I am not sure if this is a safe change.

reframe/reframe/frontend/cli.py

Line 1364 in 2546434

testcases = [t for t in testcases if not t.check.is_fixture()]