Could not access submodule when pulling recursively with git 2.22.0.windows.1

[agmikhailov]

• I was not able to find an open or closed issue matching what I'm seeing

Setup

• Which version of Git for Windows are you using? Is it 32-bit or 64-bit?

$ git --version --build-options

D:\Git\TestBranch3>git --version --build-options
git version 2.22.0.windows.1
cpu: x86_64
built from commit: d003d728ffa6c0006da875ec6318d3f6b28a4ddb
sizeof-long: 4
sizeof-size_t: 8

• Which version of Windows are you running? Vista, 7, 8, 10? Is it 32-bit or 64-bit?

$ cmd.exe /c ver

D:\Git\TestBranch3>cmd.exe /c ver

Microsoft Windows [Version 10.0.15063]

• What options did you set as part of the installation? Or did you choose the
  defaults?

# One of the following:
> type "C:\Program Files\Git\etc\install-options.txt"
> type "C:\Program Files (x86)\Git\etc\install-options.txt"
> type "%USERPROFILE%\AppData\Local\Programs\Git\etc\install-options.txt"
$ cat /etc/install-options.txt

D:\Git\TestBranch3>type "C:\Program Files\Git\etc\install-options.txt"
Editor Option: VIM
Custom Editor Path:
Path Option: Cmd
SSH Option: OpenSSH
CURL Option: OpenSSL
CRLF Option: CRLFAlways
Bash Terminal Option: MinTTY
Performance Tweaks FSCache: Enabled
Use Credential Manager: Disabled
Enable Symlinks: Disabled
Enable Builtin Interactive Add: Disabled

• Any other interesting things about your environment that might be related
  to the issue you're seeing?

This issue causes if there is submodule in submodule.

Workaround: downgrade git to version 2.20.1.

The same issue was described here:
https://gitlab.com/gitlab-org/gitlab/issues/27287

Details

• Which terminal/shell are you running Git from? e.g Bash/CMD/PowerShell/other

CMD

• What commands did you run to trigger this issue? If you can provide a
  Minimal, Complete, and Verifiable example
  this will help us understand the issue.

git.exe clone --progress -v "ssh://git@git.xxxxx.xxx:7999/xxxxxx/xxxxx.git" "D:\Git\TestBranch"
cd TestBranch
git submodule init
git submodule update
git pull --recurse-submodules=true

• What did you expect to occur after running these commands?

Successful update.

• What actually happened instead?

D:\Git\TestBranch>git pull --recurse-submodules=true
remote: Counting objects: 1, done.
remote: Total 1 (delta 0), reused 0 (delta 0)
Unpacking objects: 100% (1/1), done.
From ssh://git.xxxxx.xxx:7999/xxxx/xxxx
078bad1772..e02f419139 master -> origin/master
Fetching submodule xxxx/xxxx/xxxx
Could not access submodule 'xxxxx'Could not access submodule 'xxxxx'Fetching submodule xxxxxx/xxxxx/xxxxxxx/xxxxxx/xxxxx
Fetching submodule xxxxxx/xxxxx/xxxxx/xxxxxx
Could not access submodule 'xxxxx'Could not access submodule 'xxxxx'Fetching submodule xxxxx/xxxxxx/xxxx
Fetching submodule xxxxx/xxxxx/xxxxx/xxxxx
Could not access submodule 'xxxxx'Could not access submodule 'xxxxxx'

• If the problem was occurring with a specific repository, can you provide the
  URL to that repository to help us with testing?

No, it is our inner repository.

[dscho]

Could not access submodule 'xxxxx'

Sounds like you don't have access to that submodule.

Sorry, with such little information to go on, there is very, very little we can do over here.

You could of course try to access that git.xxxxx.xxx:7999 directly, via ssh -v -v -v git.xxxxx.xxx:7999.

[agmikhailov]

We definitely have access to the submodule of submodule and can update it directly on submodule level.

[dscho]

We definitely have access to the submodule of submodule and can update it directly on submodule level.

Git disagrees. To figure out why, write a custom script that calls ssh -v -v -v -v -v "$@" and wire it up via core.sshCommand.

[agmikhailov]

Please provide example of command you need to be executed.
Also, if I use this command it works, all submodules are updated:
git.exe clone --progress --recursive -v "ssh://git@git.xxxx.xxxx:7999/xxxxx/xxxx.git" "D:\Git\TestBranch
(there is "--recursive" option). So, it is not access problem.

[dscho]

Could not access submodule 'xxxxx'Could not access submodule 'xxxxx' [...]

I jumped to conclusions about the "Access denied", as that usually comes from remote repositories denying access. However, in this case, this error seems to be produced by the fetch orchestrator itself.

In particular, it seems that the get_submodule_repo_for() function unexpectedly does not return anything.

My best guess is that the same submodule is somehow fetched twice, in parallel, and those two fetch jobs stumble over each others' toes.

I can still only guess because the lack of any sort of reproducer, combined with the censoring of even the submodule names makes it exquisitely difficult to be of any assistance.

[agmikhailov]

My best guess is that the same submodule is somehow fetched twice, in parallel, and those two fetch jobs stumble over each others' toes.

It think it is something else. Please use my simple test repository to reproduce the problem:
https://github.com/agmikhailov/example2361.git

It is just an empty repository with one submodule (https://github.com/agmikhailov/example2361M1.git) and one submodule of submodule (https://github.com/agmikhailov/example2361M2.git):

git.exe clone --progress -v "https://github.com/agmikhailov/example2361.git" "D:\Git\example"
Cloning into 'D:\Git\example'...
POST git-upload-pack (175 bytes)
remote: Enumerating objects: 13, done.
remote: Counting objects: 100% (13/13), done.
remote: Compressing objects: 100% (7/7), done.
remote: Total 13 (delta 2), reused 12 (delta 1), pack-reused 0
Unpacking objects: 100% (13/13), done.

d:
cd D:\Git\example

git submodule init
Submodule 'example2361M1' (https://github.com/agmikhailov/example2361M1.git) registered for path 'example2361M1'

git submodule update
Cloning into 'D:/Git/example/example2361M1'...
Submodule path 'example2361M1': checked out '6a9be24a1c0ebd44d91ae4dcf1fd62580b936540'

git pull --recurse-submodules=true
Fetching submodule example2361M1
Could not access submodule 'example2361M2'

Also note, that git version 2.20.1 does not have this problem.

[dscho]

Thank you for this MCVE. That looks a lot better than what was provided in the initial report.

[dscho]

I investigated a little bit (wow, was it already 3 hours? Where does the time go when you hunt bugs...), and it turns out that this is not a Windows-specific problem. It happens on Linux, too.

Seeing as the Git for Windows project's maintainer (read: me) is already overwhelmed with the Windows-specific bugs, may I ask you to report this on the Git mailing list (please make sure that your mail is plain-text only, no HTML, not even alternative, else the mailing list will reject your mail)?

It should contain this script (which is essentially a Unix shell script version of what you wrote) to reproduce the problem:

git clone https://github.com/agmikhailov/example2361.git
cd example2361/
git submodule init
git submodule update
git pull --recurse-submodules=true

Over here, the output of the last command reads exactly like yours:

$ git --recurse-submodules=true
Fetching submodule example2361M1
Could not access submodule 'example2361M2'

[agmikhailov]

I sent report to the Git mailing list.

[agmikhailov]

Answer from the Git mailing list:

===================

"Could not access submodule" error when pulling recursively with Git 2.22.0.
This issue causes if there is submodule in submodule.

Please use my simple test repository to reproduce the problem:
https://github.com/agmikhailov/example2361.git

It is just an empty repository with one submodule
(https://github.com/agmikhailov/example2361M1.git)
and one submodule of submodule (https://github.com/agmikhailov/example2361M2.git):

git clone https://github.com/agmikhailov/example2361.git
cd example2361/
git submodule init

According to the docs of 'git submodule init', it "initializes the submodules recorded in the index". Therefore, it can only initialize the submodule in 'example2361M1', because at this point we have no idea that there is a nested submodule in there.

$ git submodule init
Submodule 'example2361M1' (https://github.com/agmikhailov/example2361M1.git) registered for path 'example2361M1'

git submodule update

This command clones 'example2361M1':

$ git submodule update --recursive
Cloning into '/tmp/example2361/example2361M1'...
Submodule path 'example2361M1': checked out '6a9be24a1c0ebd44d91ae4dcf1fd62580b936540'

Only at this point can we finally see that there is a nested submodule, and can initialize and clone it with:

$ cd example2361M1
$ git submodule init
Submodule 'example2361M2' (https://github.com/agmikhailov/example2361M2.git) registered for path 'example2361M2'
$ git submodule update
Cloning into '/tmp/example2361/example2361M1/example2361M2'...
Submodule path 'example2361M2': checked out '9ed39cf1fe0a8cf34e72d2e7ebff1ea9d4a63ac1'

git pull --recurse-submodules=true

And after that:

$ cd ../..
$ git pull --recurse-submodules=true
Fetching submodule example2361M1
Fetching submodule example2361M1/example2361M2
Already up to date.

ACTUAL RESULT

"git --recurse-submodules=true" command fails with message "Could not access submodule":

$ git --recurse-submodules=true
Fetching submodule example2361M1
Could not access submodule 'example2361M2'

EXPECTED RESULT

All submodules are successfully updated by "git --recurse-submodules=true" command.

ADDITIONAL INFORMATION

Git version 2.20.1 does not have this problem.
So we had to downgrade Git to work with submodules.

The behavior was indeed different with v2.20.1, but that version didn't show the behavior you expected. When running your commands with v2.20.1 I get:

$ ~/src/git/bin-wrappers/git pull --recurse-submodules=true
Fetching submodule example2361M1
Already up to date.
$ find example2361M1/example2361M2/
example2361M1/example2361M2/

So while that 'git pull' didn't error out, it didn't even look at the nested submodule, which is still uninitialized and empty.

===================

So it is not a bug. Old Git version just does not out the error in this case.
Sorry and thank you for collaboration.

[dscho]

Thank you for taking this to the mailing list. This was the reply: https://public-inbox.org/git/20191023100449.GH4348@szeder.dev/

Closing here, as the conversation will (hopefully) continue there.