Windows Error: install_cmdstan

[brianperdomo]

Describe the bug

I'm attempting a fresh installation of cmdstanr on Windows and I get the following error:
(FYI: The overwrite is because I tried again after the first error)

cmdstanr::install_cmdstan(overwrite = TRUE)

The C++ toolchain required for CmdStan is setup properly!
trying URL 'https://api.github.com/repos/stan-dev/cmdstan/releases/latest'
downloaded 15 KB

* Latest CmdStan release is v2.35.0
* Installing CmdStan v2.35.0 in C:\Users\brian/.cmdstan/cmdstan-2.35.0
* Downloading cmdstan-2.35.0.tar.gz from GitHub...
* Removing the existing installation of CmdStan...
trying URL 'https://github.com/stan-dev/cmdstan/releases/download/v2.35.0/cmdstan-2.35.0.tar.gz'
Content type 'application/octet-stream' length 51682840 bytes (49.3 MB)
downloaded 49.3 MB

* Download complete
* Unpacking archive...
Error in file(file, ifelse(append, "a", "w")) : 
  cannot open the connection
In addition: Warning message:
In file(file, ifelse(append, "a", "w")) :
  cannot open file 'C:\Users\brian/.cmdstan/cmdstan-2.35.0/make/local': No such file or directory

I checked that path and the problem is that the actual path should be "C:\Users\brian\.cmdstsan\cmdstan-2.35.0\cmdstan-2.35.0\make\local"

I've tried to find ways to get it to point to the right directory, but I'm at a loss.

To Reproduce
cmdstan::install_cmdstan()

Expected behavior
Either the archive needs to be unpacked in a way that matches the expected cmdstan_path() without the duplication of cmdstan-2.35.0 or cmdstan_path() should point to the path with the duplicate.... probably the former makes more sense.

Operating system
Windows 11

CmdStanR version number

tried with the version 0.8.1 from CRAN and the development version from github

Additional context
R 4.4.1, Rtools44

[jgabry]

So is the difference in the paths just the slashes? I'm not a Windows user, so maybe @rok-cesnovar or @andrjohns will have an idea about why that's happening.

I checked that path and the problem is that the actual path should be "C:\Users\brian.cmdstsan\cmdstan-2.35.0\cmdstan-2.35.0\make\local"

I assume .cmdstsan is a typo or do you really have an extra "s" in the path? That would definitely cause a problem, but guessing that's just a typo.

[jgabry]

Oh actually I see there’s two instances of cmdstan-2.35.0 in the path that you say is the correct one. That’s strange. Do you know why that’s the case?

[brianperdomo]

Sorry, the .cmdstsan was a typo. The issue is the duplication of the cmdstan-2.35.0 directory in the path.

I have not been able to figure out the issue. I'll try to take a look this weekend to see if I can find other clues. Are there any settings or other diagnostics I should check that could be informative?

[jgabry]

I don’t think I’ve seen this particular issue with the path before. A couple things to try if you’re not able to figure out why that’s happening:

• Delete the entire .cmdstan folder. Then rerun install_cmdstan(). Maybe you tried this already, if not worth a try.

• download CmdStan yourself and unpack it yourself. Then point cmdstanr to the location using set_cmdstan_path() and then run rebuild_cmdstan().

[brianperdomo]

Deleting .cmdstan and repeating the install function did not work, but I was able to unpack and rebuild cmdstan like you suggested. Thanks!

[jgabry]

I'm still not sure what the source of the problem is, but glad you got it running successfully using that workaround!

[svethitov]

Hi @jgabry,

I also managed to make it work with the workaround.

It appears that the problem is with the utils::untar function call at R/install.R. Apparently, it is not correctly executing with the --strip-components option on Windows 11. Because of that you keep the parent folder from the archive and you get this nested cmdstan-2.35.0/cmdstan-2.35.0 structure.

You can reproduce it by simply trying to untar the release manually:

utils::untar(
  "cmdstan-2.35.0.tar.gz",
  exdir = "cmdstan-2.35.0",
  extras = "--strip-components 1"
)

And the reason why it is not correctly executing is that Sys.getenv("TAR") is equal to "internal" by default on my system. Apparently, on the Windows 11 systems as a whole. However, this leads utils::untar to default to calling untar2(tarfile, files, list, exdir, restore_times). As you can see from this call it is not taking the extra parameters (extras) into account.
You can solve the issue by setting the tar executable (parameter tar of utils::untar) to tar.exe, which should be installed on Windows.

utils::untar(
  "cmdstan-2.35.0.tar.gz",
  exdir = "extract_dir_11",
  extras = "--strip-components 1",
  tar = "tar.exe"
)

This last call worked for me and it is correctly passing extra options to the unzipping, which is producing the correct folder structure.

So I suppose something like:

utils::untar(
  dest_file,
  exdir = dir_cmdstan,
  extras = "--strip-components 1",
  tar = if (os_is_windows()) "tar.exe" else Sys.getenv("TAR")
)

will do the trick.

However, another workaround is just to change the "TAR" environmental variable prior to running install_cmdstan(). I tried it and it work.

[jgabry]

@svethitov Thanks for these details, super helpful! I just created #1031 using your suggestion.

[WardBrian]

Some notes:

Sys.getenv("TAR") is equal to "internal" by default on my system. Apparently, on the Windows 11 systems as a whole

I think this is almost certainly some R funny business, TAR is not one of the environment variables that the operating system itself sets by default. "internal" also happens to be the exact special string that untar expects when requesting to not use an external tar program

tar.exe, which should be installed on Windows.

This is only true for relatively recent installations (circa April 2018)

BSD TAR.exe was added to Windows 10 (1803) from build 17063 or later.

Because it's BSD, I think it also might not support the --strip-components flag.
It does sound like Rtools includes a tar.exe, which would work as long as it appears higher in the PATH.

---

I guess one question @jgabry is why doesn't exdir just point to the parent directory (without the cmdstan_vXXX)? That would allow you to drop the --strip-components flag entirely, and then it wouldn't matter which tar implementation gets used, including internal. That way the testing/verifying the change is platform independent

[jgabry]

Thanks @WardBrian.

I guess one question @jgabry is why doesn't exdir just point to the parent directory (without the cmdstan_vXXX)? That would allow you to drop the --strip-components flag entirely, and then it wouldn't matter which tar implementation gets used, including internal. That way the testing/verifying the change is platform independent

That's a good question, I'm not sure why (I think @rok-cesnovar would know, perhaps there was a specific reason). In other words, you think we can just use

utils::untar(dest_file, exdir = dir)

where dest_file is the path to the tar.gz file and dir is either the default directory (".cmdstan" directory within the user's home directory) or a directory specified by the user. Is that right? If that works in general that sounds like a good idea.