Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add hpctestlib to the $PYTHONPATH in ReFrame 4.2.0 easyconfig #18320

Merged
merged 3 commits into from
Jul 20, 2023

Conversation

satishskamath
Copy link
Contributor

No description provided.

@casparvl
Copy link
Contributor

@boegelbot please test @ generoso

@boegelbot
Copy link
Collaborator

@casparvl: Request for testing this PR well received on login1

PR test command 'EB_PR=18320 EB_ARGS= EB_CONTAINER= /opt/software/slurm/bin/sbatch --job-name test_PR_18320 --ntasks=4 ~/boegelbot/eb_from_pr_upload_generoso.sh' executed!

  • exit code: 0
  • output:
Submitted batch job 11260

Test results coming soon (I hope)...

- notification for comment with ID 1640212967 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

@casparvl
Copy link
Contributor

@boegelbot please test @ jsc-zen2

@boegelbot
Copy link
Collaborator

@casparvl: Request for testing this PR well received on jsczen2l1.int.jsc-zen2.easybuild-test.cluster

PR test command 'EB_PR=18320 EB_ARGS= /opt/software/slurm/bin/sbatch --mem-per-cpu=4000M --job-name test_PR_18320 --ntasks=8 ~/boegelbot/eb_from_pr_upload_jsc-zen2.sh' executed!

  • exit code: 0
  • output:
Submitted batch job 3000

Test results coming soon (I hope)...

- notification for comment with ID 1640227414 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

@boegelbot
Copy link
Collaborator

Test report by @boegelbot
FAILED
Build succeeded for 0 out of 1 (1 easyconfigs in total)
jsczen2c1.int.jsc-zen2.easybuild-test.cluster - Linux Rocky Linux 8.5, x86_64, AMD EPYC 7742 64-Core Processor (zen2), Python 3.6.8
See https://gist.github.com/boegelbot/bb23cc6b5f81f76db5cf75bc8cf6ced9 for a full test report.

@casparvl
Copy link
Contributor

@boegelbot please test @ generoso

@boegelbot
Copy link
Collaborator

@casparvl: Request for testing this PR well received on login1

PR test command 'EB_PR=18320 EB_ARGS= EB_CONTAINER= /opt/software/slurm/bin/sbatch --job-name test_PR_18320 --ntasks=4 ~/boegelbot/eb_from_pr_upload_generoso.sh' executed!

  • exit code: 0
  • output:
Submitted batch job 11261

Test results coming soon (I hope)...

- notification for comment with ID 1640244951 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

@boegelbot
Copy link
Collaborator

Test report by @boegelbot
FAILED
Build succeeded for 0 out of 1 (1 easyconfigs in total)
cns1 - Linux Rocky Linux 8.5, x86_64, Intel(R) Xeon(R) CPU E5-2667 v3 @ 3.20GHz (haswell), Python 3.6.8
See https://gist.github.com/boegelbot/a8023da0defa7c46316a1a1e9be05e9d for a full test report.

@casparvl
Copy link
Contributor

Confusing, in #17873 the tests were still succesful. I don't know why, I don't see the difference (it cant reallly be your one-line change, that much is clear).

Co-authored-by: Simon Branford <4967+branfosj@users.noreply.github.com>
@casparvl
Copy link
Contributor

@boegelbot please test @ generoso

@boegelbot
Copy link
Collaborator

@casparvl: Request for testing this PR well received on login1

PR test command 'EB_PR=18320 EB_ARGS= EB_CONTAINER= /opt/software/slurm/bin/sbatch --job-name test_PR_18320 --ntasks=4 ~/boegelbot/eb_from_pr_upload_generoso.sh' executed!

  • exit code: 0
  • output:
Submitted batch job 11263

Test results coming soon (I hope)...

- notification for comment with ID 1640458821 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

@casparvl
Copy link
Contributor

@boegelbot please test @ jsc-zen2

@boegelbot
Copy link
Collaborator

Test report by @boegelbot
SUCCESS
Build succeeded for 1 out of 1 (1 easyconfigs in total)
cns1 - Linux Rocky Linux 8.5, x86_64, Intel(R) Xeon(R) CPU E5-2667 v3 @ 3.20GHz (haswell), Python 3.6.8
See https://gist.github.com/boegelbot/f23a91cffff440780216bf28999a5a6a for a full test report.

@satishskamath
Copy link
Contributor Author

Thanks @branfosj

@casparvl
Copy link
Contributor

@boegelbot please test @ jsc-zen2

@boegelbot
Copy link
Collaborator

@casparvl: Request for testing this PR well received on jsczen2l1.int.jsc-zen2.easybuild-test.cluster

PR test command 'EB_PR=18320 EB_ARGS= /opt/software/slurm/bin/sbatch --mem-per-cpu=4000M --job-name test_PR_18320 --ntasks=8 ~/boegelbot/eb_from_pr_upload_jsc-zen2.sh' executed!

  • exit code: 0
  • output:
Submitted batch job 3001

Test results coming soon (I hope)...

- notification for comment with ID 1640484806 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

@boegelbot
Copy link
Collaborator

Test report by @boegelbot
SUCCESS
Build succeeded for 1 out of 1 (1 easyconfigs in total)
jsczen2c1.int.jsc-zen2.easybuild-test.cluster - Linux Rocky Linux 8.5, x86_64, AMD EPYC 7742 64-Core Processor (zen2), Python 3.6.8
See https://gist.github.com/boegelbot/6d954d8d16e81e8376d6a505ddc876ff for a full test report.

@casparvl
Copy link
Contributor

Ok, so it builds, which is nice, but:

$ python3 -c "import hpctestlib"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'hpctestlib'

And this makes sense, because:

$ ls -al /mnt/shared/home/casparvl/.local/easybuild/software/ReFrame/4.2.0/hpctestlib
total 32
drwxr-xr-x.  8 casparvl users 6144 Apr 21 20:07 .
drwxr-xr-x. 11 casparvl users 6144 Jul 18 16:21 ..
drwxr-xr-x.  3 casparvl users 6144 Apr 21 20:07 data_analytics
drwxr-xr-x.  3 casparvl users 6144 Apr 21 20:07 interactive
drwxr-xr-x.  4 casparvl users 6144 Apr 21 20:07 microbenchmarks
drwxr-xr-x.  4 casparvl users 6144 Apr 21 20:07 ml
drwxr-xr-x.  3 casparvl users 6144 Apr 21 20:07 python
drwxr-xr-x.  4 casparvl users 6144 Apr 21 20:07 sciapps

So, either the top level prefix /mnt/shared/home/casparvl/.local/easybuild/software/ReFrame/4.2.0 should be added to the PYTHONPATH (which is weird and dirty), or we should put the hpctestlib in a subdir.

Actually I think the cleanest solution is to install hpctestlib in <prefix>/lib/python<python_maj>.<python_min>/site-packages, along with reframe itself. This is already on the pythonpath, and will allow for an import hpctestlib. It means you can actually remove the addition to the PYTHONPATH...

@satishskamath
Copy link
Contributor Author

Or we can move the hpctestlib directory to the external directory for now.

@casparvl
Copy link
Contributor

Sure, but isn't that kind of odd? It's not really external - it's part of ReFrame :D

@casparvl
Copy link
Contributor

@boegelbot please test @ generoso

@boegelbot
Copy link
Collaborator

@casparvl: Request for testing this PR well received on login1

PR test command 'EB_PR=18320 EB_ARGS= EB_CONTAINER= /opt/software/slurm/bin/sbatch --job-name test_PR_18320 --ntasks=4 ~/boegelbot/eb_from_pr_upload_generoso.sh' executed!

  • exit code: 0
  • output:
Submitted batch job 11264

Test results coming soon (I hope)...

- notification for comment with ID 1641017931 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

@boegelbot
Copy link
Collaborator

Test report by @boegelbot
SUCCESS
Build succeeded for 1 out of 1 (1 easyconfigs in total)
cns1 - Linux Rocky Linux 8.5, x86_64, Intel(R) Xeon(R) CPU E5-2667 v3 @ 3.20GHz (haswell), Python 3.6.8
See https://gist.github.com/boegelbot/999fffb3895d32f5ad076fc60f9afc22 for a full test report.

@satishskamath
Copy link
Contributor Author

@casparvl Can you launch the jsc-zen2 test via @boegelbot ?

@boegelbot
Copy link
Collaborator

@satishskamath: I noticed your comment, but I only dance when @akesandgren or @bartoldeman or @bedroge or @boegel or @branfosj or @casparvl or @jfgrimm or @lexming or @Micket or @migueldiascosta or @ocaisa or @SebastianAchilles or @smoors or @verdurin or @robert-mijakovic or @deniskristak or @ItIsI-Orient or @PetrKralCZ or @sassy-crick tells me (for now), I'm sorry...

- notification for comment with ID 1642848817 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

@casparvl
Copy link
Contributor

@boegelbot please test @ jsc-zen2

@boegelbot
Copy link
Collaborator

@casparvl: Request for testing this PR well received on jsczen2l1.int.jsc-zen2.easybuild-test.cluster

PR test command 'EB_PR=18320 EB_ARGS= /opt/software/slurm/bin/sbatch --mem-per-cpu=4000M --job-name test_PR_18320 --ntasks=8 ~/boegelbot/eb_from_pr_upload_jsc-zen2.sh' executed!

  • exit code: 0
  • output:
Submitted batch job 3004

Test results coming soon (I hope)...

- notification for comment with ID 1643553023 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

@boegelbot
Copy link
Collaborator

Test report by @boegelbot
SUCCESS
Build succeeded for 1 out of 1 (1 easyconfigs in total)
jsczen2c1.int.jsc-zen2.easybuild-test.cluster - Linux Rocky Linux 8.5, x86_64, AMD EPYC 7742 64-Core Processor (zen2), Python 3.6.8
See https://gist.github.com/boegelbot/8182d301626cc10f43756fc277729429 for a full test report.

@satishskamath
Copy link
Contributor Author

SUCCESS

[satishk@tcn3 ~]$ eblocalinstall --from-pr=18320
== Temporary log file in case of crash /scratch-shared/satishk/eb-4qkzi118/easybuild-u2hs4i_a.log
== processing EasyBuild easyconfig /scratch-shared/satishk/eb-4qkzi118/files_pr18320/r/ReFrame/ReFrame-4.2.0.eb
== building and installing ReFrame/4.2.0...
== fetching files...
== creating build dir, resetting environment...
== ... (took 1 min 12 secs)
== unpacking...
== patching...
== preparing...
== configuring...
== building...
== testing...
== installing...
== ... (took 25 secs)
== taking care of extensions...
== installing extension pip 21.3.1 (1/3)...
==      configuring...
==      building...
==      testing...
==      installing...
==      ... (took 5 secs)
== installing extension wheel 0.37.1 (2/3)...
==      configuring...
==      building...
==      testing...
==      installing...
==      ... (took 1 secs)
== installing extension reframe 4.2.0 (3/3)...
==      configuring...
==      building...
==      testing...
==      installing...
==      ... (took 45 secs)
== ... (took 53 secs)
== restore after iterating...
== postprocessing...
== sanity checking...
== ... (took 7 secs)
== cleaning up...
== ... (took 1 secs)
== creating module...
== permissions...
== packaging...
== COMPLETED: Installation ended successfully (took 2 mins 43 secs)
== Results of the build can be found in the log file(s) /home/satishk/.local/easybuild/RHEL8/2022/software/ReFrame/4.2.0/easybuild/easybuild-ReFrame-4.2.0-20230720.150419.log
== Build succeeded for 1 out of 1
== Temporary log file(s) /scratch-shared/satishk/eb-4qkzi118/easybuild-u2hs4i_a.log* have been removed.
== Temporary directory /scratch-shared/satishk/eb-4qkzi118 has been removed.
[satishk@tcn3 ~]$ module list

Currently Loaded Modules:
  1) 2022   2) EasyBuild/4.8.0   3) eb/4.8.0

 

[satishk@tcn3 ~]$ module load ReFrame/4.2.0 
[satishk@tcn3 ~]$ python
bash: python: command not found
[satishk@tcn3 ~]$ python3
Python 3.6.8 (default, Jun 20 2023, 09:53:17) 
[GCC 8.5.0 20210514 (Red Hat 8.5.0-10)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import reframe
>>> import hpctestlib
>>> 

Copy link
Contributor

@casparvl casparvl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lgtm!

@casparvl casparvl added this to the next release (4.8.1?) milestone Jul 20, 2023
@casparvl
Copy link
Contributor

Going in, thanks @satishskamath!

@casparvl casparvl merged commit d28c673 into easybuilders:develop Jul 20, 2023
5 checks passed
@boegel boegel changed the title Adding hpctestlib to the PYTHONPATH in ReFrame 4.2.0 add hpctestlib to the $PYTHONPATH in ReFrame 4.2.0 easyconfig Aug 2, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants