add `hpctestlib` to the `$PYTHONPATH` in ReFrame 4.2.0 easyconfig #18320

satishskamath · 2023-07-17T15:52:32Z

No description provided.

casparvl · 2023-07-18T13:17:02Z

@boegelbot please test @ generoso

boegelbot · 2023-07-18T13:20:10Z

@casparvl: Request for testing this PR well received on login1

PR test command 'EB_PR=18320 EB_ARGS= EB_CONTAINER= /opt/software/slurm/bin/sbatch --job-name test_PR_18320 --ntasks=4 ~/boegelbot/eb_from_pr_upload_generoso.sh' executed!

exit code: 0
output:

Submitted batch job 11260

Test results coming soon (I hope)...

- notification for comment with ID 1640212967 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

casparvl · 2023-07-18T13:26:15Z

@boegelbot please test @ jsc-zen2

boegelbot · 2023-07-18T13:27:09Z

@casparvl: Request for testing this PR well received on jsczen2l1.int.jsc-zen2.easybuild-test.cluster

PR test command 'EB_PR=18320 EB_ARGS= /opt/software/slurm/bin/sbatch --mem-per-cpu=4000M --job-name test_PR_18320 --ntasks=8 ~/boegelbot/eb_from_pr_upload_jsc-zen2.sh' executed!

exit code: 0
output:

Submitted batch job 3000

Test results coming soon (I hope)...

- notification for comment with ID 1640227414 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

boegelbot · 2023-07-18T13:28:28Z

Test report by @boegelbot
FAILED
Build succeeded for 0 out of 1 (1 easyconfigs in total)
jsczen2c1.int.jsc-zen2.easybuild-test.cluster - Linux Rocky Linux 8.5, x86_64, AMD EPYC 7742 64-Core Processor (zen2), Python 3.6.8
See https://gist.github.com/boegelbot/bb23cc6b5f81f76db5cf75bc8cf6ced9 for a full test report.

casparvl · 2023-07-18T13:37:20Z

@boegelbot please test @ generoso

boegelbot · 2023-07-18T13:40:10Z

@casparvl: Request for testing this PR well received on login1

PR test command 'EB_PR=18320 EB_ARGS= EB_CONTAINER= /opt/software/slurm/bin/sbatch --job-name test_PR_18320 --ntasks=4 ~/boegelbot/eb_from_pr_upload_generoso.sh' executed!

exit code: 0
output:

Submitted batch job 11261

Test results coming soon (I hope)...

- notification for comment with ID 1640244951 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

boegelbot · 2023-07-18T13:41:42Z

Test report by @boegelbot
FAILED
Build succeeded for 0 out of 1 (1 easyconfigs in total)
cns1 - Linux Rocky Linux 8.5, x86_64, Intel(R) Xeon(R) CPU E5-2667 v3 @ 3.20GHz (haswell), Python 3.6.8
See https://gist.github.com/boegelbot/a8023da0defa7c46316a1a1e9be05e9d for a full test report.

casparvl · 2023-07-18T14:24:50Z

Confusing, in #17873 the tests were still succesful. I don't know why, I don't see the difference (it cant reallly be your one-line change, that much is clear).

easybuild/easyconfigs/r/ReFrame/ReFrame-4.2.0.eb

Co-authored-by: Simon Branford <4967+branfosj@users.noreply.github.com>

casparvl · 2023-07-18T15:32:00Z

@boegelbot please test @ generoso

boegelbot · 2023-07-18T15:35:09Z

@casparvl: Request for testing this PR well received on login1

PR test command 'EB_PR=18320 EB_ARGS= EB_CONTAINER= /opt/software/slurm/bin/sbatch --job-name test_PR_18320 --ntasks=4 ~/boegelbot/eb_from_pr_upload_generoso.sh' executed!

exit code: 0
output:

Submitted batch job 11263

Test results coming soon (I hope)...

- notification for comment with ID 1640458821 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

casparvl · 2023-07-18T15:35:29Z

@boegelbot please test @ jsc-zen2

boegelbot · 2023-07-18T15:36:58Z

Test report by @boegelbot
SUCCESS
Build succeeded for 1 out of 1 (1 easyconfigs in total)
cns1 - Linux Rocky Linux 8.5, x86_64, Intel(R) Xeon(R) CPU E5-2667 v3 @ 3.20GHz (haswell), Python 3.6.8
See https://gist.github.com/boegelbot/f23a91cffff440780216bf28999a5a6a for a full test report.

satishskamath · 2023-07-18T15:38:37Z

Thanks @branfosj

casparvl · 2023-07-18T15:45:22Z

@boegelbot please test @ jsc-zen2

boegelbot · 2023-07-18T15:47:08Z

@casparvl: Request for testing this PR well received on jsczen2l1.int.jsc-zen2.easybuild-test.cluster

PR test command 'EB_PR=18320 EB_ARGS= /opt/software/slurm/bin/sbatch --mem-per-cpu=4000M --job-name test_PR_18320 --ntasks=8 ~/boegelbot/eb_from_pr_upload_jsc-zen2.sh' executed!

exit code: 0
output:

Submitted batch job 3001

Test results coming soon (I hope)...

- notification for comment with ID 1640484806 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

boegelbot · 2023-07-18T15:48:27Z

Test report by @boegelbot
SUCCESS
Build succeeded for 1 out of 1 (1 easyconfigs in total)
jsczen2c1.int.jsc-zen2.easybuild-test.cluster - Linux Rocky Linux 8.5, x86_64, AMD EPYC 7742 64-Core Processor (zen2), Python 3.6.8
See https://gist.github.com/boegelbot/6d954d8d16e81e8376d6a505ddc876ff for a full test report.

casparvl · 2023-07-18T16:33:40Z

Ok, so it builds, which is nice, but:

$ python3 -c "import hpctestlib"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'hpctestlib'

And this makes sense, because:

$ ls -al /mnt/shared/home/casparvl/.local/easybuild/software/ReFrame/4.2.0/hpctestlib
total 32
drwxr-xr-x.  8 casparvl users 6144 Apr 21 20:07 .
drwxr-xr-x. 11 casparvl users 6144 Jul 18 16:21 ..
drwxr-xr-x.  3 casparvl users 6144 Apr 21 20:07 data_analytics
drwxr-xr-x.  3 casparvl users 6144 Apr 21 20:07 interactive
drwxr-xr-x.  4 casparvl users 6144 Apr 21 20:07 microbenchmarks
drwxr-xr-x.  4 casparvl users 6144 Apr 21 20:07 ml
drwxr-xr-x.  3 casparvl users 6144 Apr 21 20:07 python
drwxr-xr-x.  4 casparvl users 6144 Apr 21 20:07 sciapps

So, either the top level prefix /mnt/shared/home/casparvl/.local/easybuild/software/ReFrame/4.2.0 should be added to the PYTHONPATH (which is weird and dirty), or we should put the hpctestlib in a subdir.

Actually I think the cleanest solution is to install hpctestlib in <prefix>/lib/python<python_maj>.<python_min>/site-packages, along with reframe itself. This is already on the pythonpath, and will allow for an import hpctestlib. It means you can actually remove the addition to the PYTHONPATH...

satishskamath · 2023-07-18T16:39:54Z

Or we can move the hpctestlib directory to the external directory for now.

PYTHONPATH.

casparvl · 2023-07-18T18:10:27Z

Sure, but isn't that kind of odd? It's not really external - it's part of ReFrame :D

casparvl · 2023-07-18T21:29:46Z

@boegelbot please test @ generoso

boegelbot · 2023-07-18T21:30:09Z

@casparvl: Request for testing this PR well received on login1

PR test command 'EB_PR=18320 EB_ARGS= EB_CONTAINER= /opt/software/slurm/bin/sbatch --job-name test_PR_18320 --ntasks=4 ~/boegelbot/eb_from_pr_upload_generoso.sh' executed!

exit code: 0
output:

Submitted batch job 11264

Test results coming soon (I hope)...

- notification for comment with ID 1641017931 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

boegelbot · 2023-07-18T21:32:01Z

Test report by @boegelbot
SUCCESS
Build succeeded for 1 out of 1 (1 easyconfigs in total)
cns1 - Linux Rocky Linux 8.5, x86_64, Intel(R) Xeon(R) CPU E5-2667 v3 @ 3.20GHz (haswell), Python 3.6.8
See https://gist.github.com/boegelbot/999fffb3895d32f5ad076fc60f9afc22 for a full test report.

satishskamath · 2023-07-19T22:28:32Z

@casparvl Can you launch the jsc-zen2 test via @boegelbot ?

boegelbot · 2023-07-19T22:32:08Z

@satishskamath: I noticed your comment, but I only dance when @akesandgren or @bartoldeman or @bedroge or @boegel or @branfosj or @casparvl or @jfgrimm or @lexming or @Micket or @migueldiascosta or @ocaisa or @SebastianAchilles or @smoors or @verdurin or @robert-mijakovic or @deniskristak or @ItIsI-Orient or @PetrKralCZ or @sassy-crick tells me (for now), I'm sorry...

- notification for comment with ID 1642848817 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

casparvl · 2023-07-20T09:04:26Z

@boegelbot please test @ jsc-zen2

boegelbot · 2023-07-20T09:07:09Z

@casparvl: Request for testing this PR well received on jsczen2l1.int.jsc-zen2.easybuild-test.cluster

PR test command 'EB_PR=18320 EB_ARGS= /opt/software/slurm/bin/sbatch --mem-per-cpu=4000M --job-name test_PR_18320 --ntasks=8 ~/boegelbot/eb_from_pr_upload_jsc-zen2.sh' executed!

exit code: 0
output:

Submitted batch job 3004

Test results coming soon (I hope)...

- notification for comment with ID 1643553023 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

boegelbot · 2023-07-20T09:08:23Z

Test report by @boegelbot
SUCCESS
Build succeeded for 1 out of 1 (1 easyconfigs in total)
jsczen2c1.int.jsc-zen2.easybuild-test.cluster - Linux Rocky Linux 8.5, x86_64, AMD EPYC 7742 64-Core Processor (zen2), Python 3.6.8
See https://gist.github.com/boegelbot/8182d301626cc10f43756fc277729429 for a full test report.

satishskamath · 2023-07-20T13:07:51Z

SUCCESS

[satishk@tcn3 ~]$ eblocalinstall --from-pr=18320
== Temporary log file in case of crash /scratch-shared/satishk/eb-4qkzi118/easybuild-u2hs4i_a.log
== processing EasyBuild easyconfig /scratch-shared/satishk/eb-4qkzi118/files_pr18320/r/ReFrame/ReFrame-4.2.0.eb
== building and installing ReFrame/4.2.0...
== fetching files...
== creating build dir, resetting environment...
== ... (took 1 min 12 secs)
== unpacking...
== patching...
== preparing...
== configuring...
== building...
== testing...
== installing...
== ... (took 25 secs)
== taking care of extensions...
== installing extension pip 21.3.1 (1/3)...
==      configuring...
==      building...
==      testing...
==      installing...
==      ... (took 5 secs)
== installing extension wheel 0.37.1 (2/3)...
==      configuring...
==      building...
==      testing...
==      installing...
==      ... (took 1 secs)
== installing extension reframe 4.2.0 (3/3)...
==      configuring...
==      building...
==      testing...
==      installing...
==      ... (took 45 secs)
== ... (took 53 secs)
== restore after iterating...
== postprocessing...
== sanity checking...
== ... (took 7 secs)
== cleaning up...
== ... (took 1 secs)
== creating module...
== permissions...
== packaging...
== COMPLETED: Installation ended successfully (took 2 mins 43 secs)
== Results of the build can be found in the log file(s) /home/satishk/.local/easybuild/RHEL8/2022/software/ReFrame/4.2.0/easybuild/easybuild-ReFrame-4.2.0-20230720.150419.log
== Build succeeded for 1 out of 1
== Temporary log file(s) /scratch-shared/satishk/eb-4qkzi118/easybuild-u2hs4i_a.log* have been removed.
== Temporary directory /scratch-shared/satishk/eb-4qkzi118 has been removed.
[satishk@tcn3 ~]$ module list

Currently Loaded Modules:
  1) 2022   2) EasyBuild/4.8.0   3) eb/4.8.0

 

[satishk@tcn3 ~]$ module load ReFrame/4.2.0 
[satishk@tcn3 ~]$ python
bash: python: command not found
[satishk@tcn3 ~]$ python3
Python 3.6.8 (default, Jun 20 2023, 09:53:17) 
[GCC 8.5.0 20210514 (Red Hat 8.5.0-10)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import reframe
>>> import hpctestlib
>>>

casparvl

Lgtm!

casparvl · 2023-07-20T14:59:30Z

Going in, thanks @satishskamath!

Adding hpctestlib to the PYTHONPATH.

af4065c

casparvl mentioned this pull request Jul 18, 2023

{devel}[system/system] ReFrame v4.2.0 #17873

Merged

branfosj reviewed Jul 18, 2023

View reviewed changes

easybuild/easyconfigs/r/ReFrame/ReFrame-4.2.0.eb Outdated Show resolved Hide resolved

Update easybuild/easyconfigs/r/ReFrame/ReFrame-4.2.0.eb

0785c0b

Co-authored-by: Simon Branford <4967+branfosj@users.noreply.github.com>

Now hpctestlib is copied to external which is included in the

718620d

PYTHONPATH.

satishskamath requested review from casparvl and branfosj July 18, 2023 17:10

casparvl approved these changes Jul 20, 2023

View reviewed changes

casparvl added this to the next release (4.8.1?) milestone Jul 20, 2023

casparvl added the bug fix label Jul 20, 2023

casparvl merged commit d28c673 into easybuilders:develop Jul 20, 2023
5 checks passed

boegel changed the title ~~Adding hpctestlib to the PYTHONPATH in ReFrame 4.2.0~~ add hpctestlib to the $PYTHONPATH in ReFrame 4.2.0 easyconfig Aug 2, 2023

boegel mentioned this pull request Sep 5, 2023

fix installation of ReFrame 4.2.0 (include hpctestlib) in EESSI pilot 2023.06 EESSI/software-layer#311

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add `hpctestlib` to the `$PYTHONPATH` in ReFrame 4.2.0 easyconfig #18320

add `hpctestlib` to the `$PYTHONPATH` in ReFrame 4.2.0 easyconfig #18320

satishskamath commented Jul 17, 2023

casparvl commented Jul 18, 2023

boegelbot commented Jul 18, 2023

casparvl commented Jul 18, 2023

boegelbot commented Jul 18, 2023

boegelbot commented Jul 18, 2023

casparvl commented Jul 18, 2023

boegelbot commented Jul 18, 2023

boegelbot commented Jul 18, 2023

casparvl commented Jul 18, 2023

casparvl commented Jul 18, 2023

boegelbot commented Jul 18, 2023

casparvl commented Jul 18, 2023

boegelbot commented Jul 18, 2023

satishskamath commented Jul 18, 2023

casparvl commented Jul 18, 2023

boegelbot commented Jul 18, 2023

boegelbot commented Jul 18, 2023

casparvl commented Jul 18, 2023

satishskamath commented Jul 18, 2023

casparvl commented Jul 18, 2023

casparvl commented Jul 18, 2023

boegelbot commented Jul 18, 2023

boegelbot commented Jul 18, 2023

satishskamath commented Jul 19, 2023

boegelbot commented Jul 19, 2023

casparvl commented Jul 20, 2023

boegelbot commented Jul 20, 2023

boegelbot commented Jul 20, 2023

satishskamath commented Jul 20, 2023

casparvl left a comment

casparvl commented Jul 20, 2023

add hpctestlib to the $PYTHONPATH in ReFrame 4.2.0 easyconfig #18320

add hpctestlib to the $PYTHONPATH in ReFrame 4.2.0 easyconfig #18320

Conversation

satishskamath commented Jul 17, 2023

casparvl commented Jul 18, 2023

boegelbot commented Jul 18, 2023

casparvl commented Jul 18, 2023

boegelbot commented Jul 18, 2023

boegelbot commented Jul 18, 2023

casparvl commented Jul 18, 2023

boegelbot commented Jul 18, 2023

boegelbot commented Jul 18, 2023

casparvl commented Jul 18, 2023

casparvl commented Jul 18, 2023

boegelbot commented Jul 18, 2023

casparvl commented Jul 18, 2023

boegelbot commented Jul 18, 2023

satishskamath commented Jul 18, 2023

casparvl commented Jul 18, 2023

boegelbot commented Jul 18, 2023

boegelbot commented Jul 18, 2023

casparvl commented Jul 18, 2023

satishskamath commented Jul 18, 2023

casparvl commented Jul 18, 2023

casparvl commented Jul 18, 2023

boegelbot commented Jul 18, 2023

boegelbot commented Jul 18, 2023

satishskamath commented Jul 19, 2023

boegelbot commented Jul 19, 2023

casparvl commented Jul 20, 2023

boegelbot commented Jul 20, 2023

boegelbot commented Jul 20, 2023

satishskamath commented Jul 20, 2023

casparvl left a comment

Choose a reason for hiding this comment

casparvl commented Jul 20, 2023

add `hpctestlib` to the `$PYTHONPATH` in ReFrame 4.2.0 easyconfig #18320

add `hpctestlib` to the `$PYTHONPATH` in ReFrame 4.2.0 easyconfig #18320