Submodules of git dependencies are not fetched

[ntucker]

Do you want to request a feature or report a bug?
Bug

What is the current behavior?
Having a git dependency with a submodule results in the submodule being an empty directory

If the current behavior is a bug, please provide the steps to reproduce.

1. Setup NPM package at a github url
2. Add submodule to said package
3. Add dependency in package.json for local project that points to the github url of said package
4. Yarn install
5. Check node_modules directory for that package and see that submodule folder is empty.

What is the expected behavior?
Submodule directories should have the contents (the cloning process should run submodule init and submodule update)

Please mention your node.js, yarn and operating system version.
Node: 7.0.0
Yarn: 0.16.1
OS: Ubuntu 14.04

[jokeyrhyme]

Here's the discussion for when they added this to npm: npm/npm#1876

[hector]

+1

Some npm packages won't work without this feature.

[DeDuckProject]

Moreover, the submodules get deleted if they are in there rightful place before running yarn

[hector]

I look forward to start using yarn but I cannot switch until this gets fixed :(

[leggomuhgreggo]

+1

[rogerkerse]

Any news on it?

[gedgaudasnikita]

While i hope this issue is getting some research invested into it, this placeholder workaround might serve as a temporary solution for all of us waiting

In package.json of a module being fetched, add a following script
"postinstall": "grep url .gitmodules | sed 's/.*= //' | while read url; do git clone $url; done"

[Qix-]

So here's the synopsis of the problem:

The Git utilities included with Yarn seem to do a git clone and then git archive to archive the repository. I've modified the clone command to include --depth=1 --recursive, but the problem is with git archive - it's been identified that it doesn't archive submodules and thus will need to have a workaround.

I'll see what I can do.

[Qix-]

Also, I highly suggest that support for remote archiving is removed. Git has terrible support for it, and it would require a lot of extra code just to get submodules working. As it stands, there isn't any code that enables any of the features, and github.com is explicitly disabled for remote archiving.

I'll make a note of this in the eventual PR and ask the owners of Yarn if it should be removed. I agree with the comment in-source:

const supportsArchiveCache: {[key: string]: boolean} = map({
  'github.com': false, // not support, doubt they will ever support it
});

Also, the code for determining if the URI supports remote archiving is extremely hackish and nasty. It'd be a delight to see it gone IMO.

try {
  await Git.spawn(['archive', `--remote=${ref.repository}`, 'HEAD', Date.now() + '']);
  throw new Error();
} catch (err) {
  const supports = err.message.indexOf('did not match any files') >= 0;
  return (supportsArchiveCache[hostname] = supports);
}

For all of the extra code overhead and this lengthy check on what appears to be every fetch, I'm not convinced the tradeoff is worth it.

Without it, we can simply do a bulk copy (or even better, a bulk hard link - but I won't write that logic 😅) between the fetched and the target directory, which would fix this issue and simplify the fetching code a bit more - and probably make cloning from a repository much faster since it requires one less request.

[andreimc]

is there any fix for this ?