remove manage externals

[jedwards4b]

Description of changes

Starting from slevis-lmwg/upd_externals_to_beta17 remove manage externals and add git-fleximod

Takes the place of #2443 and continues the discussion from there.

Specific notes

Contributors other than yourself, if any:

CTSM Issues Fixed (include github issue #):
Fixes #2536
Fixes #2537

Are answers expected to change (and if so in what way)? No

Any User Interface Changes (namelist or namelist defaults changes)? for handling externals

Does this create a need to change or add documentation? Did you do so? Yes, in-process

Testing performed, if any: regular testing + ctsm_sci

[jedwards4b]

@slevis-lmwg this branch is ready for testing. Can you kick that off and let me know if you have any issues?

[jedwards4b]

@ekluzek @slevis-lmwg want to make sure this PR is advancing this morning, I think that having Sam do the testing will be the best approach, please confirm.

[slevis-lmwg]

@jedwards4b I plan to start testing this morning.

[slevis-lmwg]

@jedwards4b I checked out your branch and proceeded as follows:

git checkout -b ctsm5.2.007/candidate jedwards4b/ctsm5.2.007/candidate
git branch
git status
git describe
git fleximod status
pip install git-fleximod
git fleximod status
git fleximod update

and got this message:
Repository has local mods, cowardly refusing to continue, fix issues or use --force to override
Do you want me to use --force to override?

[jedwards4b]

Hi Sam, The process should be:

git clone https://github.com/jedwards4b/CTSM -b ctsm5.2.007/candidate ctsm5.2.007_candidate
cd ctsm5.2.007_candidate
./bin/git-fleximod update

[jedwards4b]

To elaborate further - because the choice was made to include git-fleximod with the distribution I haven't updated the pip version in a while, it's out of date. Once we have created the cesm3_0_beta01 tag I will update the version available with pip.

[slevis-lmwg]

aux_clm fails right off the bat because it's searching for ./manage_externals/checkout_externals
I may or may not have time to fix this before my 10 am meeting...

[jedwards4b]

What is searching for manage_externals - a script outside the repo???

[ekluzek]

I added some issues that need to be handled at the top. One of which is changes to run_sys_tests so that the testing will work.

@jedwards4b we have a wrapper script to create_test in ctsm at the top level run_sys_tests that makes it easier to run our testing for CTSM.

[slevis-lmwg]

I submit the aux_clm test-suite like this:
./run_sys_tests -s aux_clm -c ctsm5.2.006 -g ctsm5.2.007
so run_sys_tests is looking for ./manage_externals/checkout_externals.

Meanwhile though...
derecho tests
PASS make black and make lint
PASS ./run_ctsm_py_tests -u and ./run_ctsm_py_tests -s
PASS ./build-namelist_test.pl
ctsm_sci IN PROGRESS
aux_clm IN PROGRESS
OK aux_clm on izumi

[jedwards4b]

@slevis-lmwg I pushed an update that should solve the problem.

[slevis-lmwg]

@jedwards4b while I wait for tests to complete, I'm working on the list in #2536
and my question to you is whether the documentation should say,
"run ./bin/git-fleximod update" or
"run git fleximod update"

[jedwards4b]

I think that it should say "run ./bin/git-fleximod update"

[ekluzek]

@jedwards4b I get the following error both on Derecho and Izumi when I try to use gitfleximod...

Below is from Izumi

./bin/git-fleximod update
Traceback (most recent call last):
  File "./bin/git-fleximod", line 8, in <module>
    sys.exit(main())
  File "/fs/cgd/data0/erik/ctsm_worktree/externals_update/.lib/git-fleximod/git_fleximod/git_fleximod.py", line 559, in main
    if not os.path.isfile(os.path.join(root_dir, file_name)):
  File "/usr/local/anaconda-3.7.0/lib/python3.7/posixpath.py", line 80, in join
    a = os.fspath(a)
TypeError: expected str, bytes or os.PathLike object, not NoneType

It's working for others, so I'm not sure what's different for me. I am using tcsh on both systems, but I also tried it from bash and got the same behavior. It didn't appear that gitfleximod used any ENV variables so I don't think it's anything there.

Let me know if you have any ideas on what to try. Thanks.

[jedwards4b]

@ekluzek python 3.7 should be new enough but please try again without your conda environment loaded. Whether or not that works - please be specific about how to reproduce this error.

[jedwards4b]

@ekluzek You have something strange going on. I don't think that the problem is your conda environment. git-fleximod is looking for a directory /fs/cgd/data0/erik/ctsm_worktree/externals_update/.git but in your case this is a file with contents:
gitdir: /fs/cgd/data0/erik/ctsm_neon/.git/worktrees/externals_update

Indicating that you have checked out ctsm as a submodule of ctsm_neon but then moved it in the directory system so that it is no longer a subdirectory of /fs/cgd/data0/erik/ctsm_neon

[jedwards4b]

I can make a modification to partially handle this case, the error with my change will be:
ERROR: No .gitmodules found in /fs/cgd/data0/erik/ctsm_neon/.git/worktrees/externals_update or any of it's parents
which I don't think does a lot to clarify what's going on. I need to understand what you are doing to get this convoluted directory structure???

[ekluzek]

@jedwards4b note I've tried both with the ctsm_pylib conda environment loaded as well as not loaded (so python 3.7.9, and 3.10.10). And I've tried the same on izumi (python 3.7.0).

I also tried in @slevis-lmwg clone and get an error message as well.

git_externals/ctsm5.2.007_candidate> ./bin/git-fleximod status
Traceback (most recent call last):
  File "/glade/work/slevis/git_externals/ctsm5.2.007_candidate/./bin/git-fleximod", line 8, in <module>
    sys.exit(main())
  File "/glade/work/slevis/git_externals/ctsm5.2.007_candidate/.lib/git-fleximod/git_fleximod/git_fleximod.py", line 586, in main
    tfails, lmods, updates = submodules_status(gitmodules, root_dir, toplevel=True)
  File "/glade/work/slevis/git_externals/ctsm5.2.007_candidate/.lib/git-fleximod/git_fleximod/git_fleximod.py", line 315, in submodules_status
    ahash = git.git_operation("status").partition("\n")[0].split()[-1]
IndexError: list index out of range
git_externals/ctsm5.2.007_candidate> python --version
Python 3.10.10
git_externals/ctsm5.2.007_candidate> pwd
/glade/work/slevis/git_externals/ctsm5.2.007_candidate

I wonder if the above has to do with permissions to his clone though?

In terms of my directory structure. I'm using

git worktree

so that I have a list of clone's, that share information between them (remote definition, remote fetch etc.). This is something that I learned from @billsacks so he uses it as well. I've found it useful, so I'd be surprised if he and I were the only ones to use it. As a user it manages the .git directory information for me, so I don't know (and don't care) what it's doing under the covers.

[ekluzek]

@jedwards4b what information does gitfleximod get from the .git subdirectory? And are there different ways to get the same thing? Depending on the .git subdirectory file structure, does mean that gitfleximod is tied to particular versions of git, and anytime git changes anything required, gitfleximod will break. So it seems desirable to not rely on anything there, if possible, unless git development views their .git subdirectory file structure as a fixed (or at least a rarely changing) feature. We do want gitfleximod to work with a wide variety of git versions, and hopefully to be stable as new versions of git become available.

[jedwards4b]

@ekluzek Please try again - I think I have corrected the issue.

[slevis-lmwg]

@jedwards4b @ekluzek I defer to your judgement on whether to rerun testing.

[jedwards4b]

@slevis-lmwg I'll be out this afternoon but hope you can merge this before I check in tomorrow morning. Thanks for all your work - oh and please update the ChangeLog.

[ekluzek]

@jedwards4b thanks for the gitfleximod update! That seems to work for me. I'm going to try more things out, but I don't envision anything to hold finishing this by tomorrow morning.

[samsrabin]

Derecho slowness gave me reason to catch this remaining reference to manage_externals (not sure if it's been mentioned elsewhere):

Command 'git -C /glade/u/home/samrabin/ctsm5.2.007_candidate submodule update --init -- cime'
from directory /glade/u/home/samrabin/ctsm5.2.007_candidate
has taken 300 seconds. It may be hanging.

The command will continue to run, but you may want to abort
manage_externals with ^C and investigate. A possible cause of hangs is
when svn or git require authentication to access a private
repository. On some systems, svn and git requests for authentication
information will not be displayed to the user. In this case, the program
will appear to hang. Ensure you can run svn and git manually and access
all repositories without entering your authentication information.

[ekluzek]

This is great both @slevis-lmwg and @jedwards4b. There are several advantages to using "git submodules" over manage_externals, so it's nice to bring this in.

@slevis-lmwg I'm marking this as approve, but I do have a few tweaks I'd like to make to the README.GITFLEXIMOD file. I think it would be easier for me to commit and push directly rather than trying to explain. Does that sound OK to you? If I don't have permissions we'll figure out how to do it.

[slevis-lmwg]

from conversation with @ekluzek
I will go ahead with the merge now.

[jedwards4b]

Great - thank you @slevis-lmwg and @ekluzek. Have a great weekend.