vs/master not building when install-option "Path Option: BashOnly" in use (probably)

[PhilipOakley]

I created a worktree to hold the origin/vs/master to compile git for windows using VS2017. Unfortunately the compilation failed. Probably because the batch file to load the vcpkg dependencies failed to find git.exe

Arguably it's probably my choice of setup (Path Option: BashOnly), but then again, maybe it should 'just work', or have a better error message on how to 'fix' the PathOption choice.

• [x ] I was not able to find an open or closed issue matching what I'm seeing

Setup

• Which version of Git for Windows are you using? Is it 32-bit or 64-bit?

$ git --version --build-options

git version 2.23.0.windows.1.21.g947f504ebe8.dirty
cpu: x86_64
built from commit: 947f504ebe8fc780465d92bde25f3b576bf2f21f
sizeof-long: 4
sizeof-size_t: 8

• Which version of Windows are you running? Vista, 7, 8, 10? Is it 32-bit or 64-bit?

$ cmd.exe /c ver

Microsoft Windows [Version 10.0.17763.775]

• What options did you set as part of the installation? Or did you choose the
  defaults?

# One of the following:
> type "C:\Program Files\Git\etc\install-options.txt"
> type "C:\Program Files (x86)\Git\etc\install-options.txt"
> type "%USERPROFILE%\AppData\Local\Programs\Git\etc\install-options.txt"
$ cat /etc/install-options.txt

Editor Option: Notepad++
Custom Editor Path:
Path Option: BashOnly
SSH Option: OpenSSH
Tortoise Option: false
CURL Option: OpenSSL
CRLF Option: CRLFAlways
Bash Terminal Option: MinTTY
Performance Tweaks FSCache: Enabled
Use Credential Manager: Enabled
Enable Symlinks: Disabled
Enable Builtin Interactive Add: Enabled

• Any other interesting things about your environment that might be related
  to the issue you're seeing?

SDK install

Details

• Which terminal/shell are you running Git from? e.g Bash/CMD/PowerShell/other

Bash, and Win10 double-click

• What commands did you run to trigger this issue? If you can provide a
  Minimal, Complete, and Verifiable example
  this will help us understand the issue.

$ git worktree add ../VisualStudioGit vs/master
Preparing worktree (checking out 'vs/master')
Updating files: 100% (3742/3742), done.
HEAD is now at 6405c8db46a Generate Visual Studio solution

Windows explorer, navigate to git.sln and double click, starting VS2017. Ignore the upgrade screen.
Build solution

1>------ Build started: Project: libgit, Configuration: Debug x64 ------
1>Fetching vcpkg in C:\git-sdk-64\usr\src\VisualStudioGit\compat\vcbuild\vcpkg
1>'git.exe' is not recognized as an internal or external command,
1>operable program or batch file.
1>C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\V140\Microsoft.CppCommon.targets(123,5): error MSB3073: The command "call "..\compat\vcbuild\vcpkg_install.bat"
1>C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\V140\Microsoft.CppCommon.targets(123,5): error MSB3073: :VCEnd" exited with code 1.
1>Done building project "libgit.vcxproj" -- FAILED.
2>------ Build started: Project: xdiff_lib, Configuration: Debug x64 ------
2>xutils.c
2>c:\git-sdk-64\usr\src\visualstudiogit\compat/mingw.h(178): fatal error C1083: Cannot open include file: 'openssl/ssl.h': No such file or directory
...

Note that git.exe is not found for this installation.

• What did you expect to occur after running these commands?

successful compilation, or at least the down load of the vcpkg and the old error ;-)

• What actually happened instead?

see above, git.exe not found.

• If the problem was occurring with a specific repository, can you provide the
  URL to that repository to help us with testing?

This here repo...

[PhilipOakley]

s/ old error ;-)/ odd error ;-)/ finger trouble.

Basically, I (think I) need to find the 'proper' method for changing the Path Option (been avoiding looking into this one), so that the cmd window (used by the .bat file) will find the right (acceptable) git.exe, even if it is the C:\Program Files\Git version...

[dscho]

Indeed, I believe that vcpkg requires git.exe to be in the PATH.

Of course, you can work around this by extending your PATH manually, to include C:\Program Files\Git\cmd. If you do that in the System Properties, the CMD interpreter opened by Visual Studio to run vcpkg.exe should be able to find git.exe.

[PhilipOakley]

Just reviewing the installer code.

first look through for 'path' and find:
https://github.com/git-for-windows/build-extra/blob/master/installer/install.iss#L1656-L1678

    (*
     * Create a custom page for modifying the environment.
     *)

    PathPage:=CreatePage(PrevPageID,'Adjusting your PATH environment','How would you like to use Git from the command line?',TabOrder,Top,Left);

    // 1st choice
    RdbPath[GP_BashOnly]:=CreateRadioButton(PathPage,'Use Git from Git Bash only','This is the most cautious choice as your PATH will not be modified at all. You will'+#13+'only be able to use the Git command line tools from Git Bash.',TabOrder,Top,Left);

    // 2nd choice
    RdbPath[GP_Cmd]:=CreateRadioButton(PathPage,'Git from the command line and also from 3rd-party software','(Recommended) This option adds only some minimal Git wrappers to your'+#13+'PATH to avoid cluttering your environment with optional Unix tools.'+#13+'You will be able to use Git from Git Bash, the Command Prompt and the Windows'+#13+'PowerShell as well as any third-party software looking for Git in PATH.',TabOrder,Top,Left);

    // 3rd choice
    RdbPath[GP_CmdTools]:=CreateRadioButton(PathPage,'Use Git and optional Unix tools from the Command Prompt','Both Git and the optional Unix tools will be added to your PATH.'+#13+'<RED>Warning: This will override Windows tools like "find" and "sort". Only'+#13+'use this option if you understand the implications.</RED>',TabOrder,Top,Left);

    // Restore the setting chosen during a previous install.
    case ReplayChoice('Path Option','Cmd') of
        'BashOnly': RdbPath[GP_BashOnly].Checked:=True;
        'Cmd': RdbPath[GP_Cmd].Checked:=True;
        'CmdTools': RdbPath[GP_CmdTools].Checked:=True;
    else
        RdbPath[GP_Cmd].Checked:=True;
    end;

now look for GP_Cmd which is

`Git from the command line and also from 3rd-party software',
'(Recommended) This option adds only some minimal Git wrappers to your'
'PATH to avoid cluttering your environment with optional Unix tools.'
'You will be able to use Git from Git Bash, the Command Prompt and the Windows'
'PowerShell as well as any third-party software looking for Git in PATH.':

https://github.com/git-for-windows/build-extra/blob/master/installer/install.iss#L2627-L2655

    // Get the current user's directories in PATH.
    EnvPath:=GetEnvStrings('PATH',IsAdminLoggedOn);

    // Modify the PATH variable as requested by the user.
    if RdbPath[GP_Cmd].Checked or RdbPath[GP_CmdTools].Checked then begin
        // First, remove the current installation directory from PATH.
        for i:=0 to GetArrayLength(EnvPath)-1 do begin
            if Pos(AppDir+'\',EnvPath[i]+'\')=1 then begin
                EnvPath[i]:='';
            end;
        end;

        i:=GetArrayLength(EnvPath);
        SetArrayLength(EnvPath,i+1);

        // List \cmd before \bin so \cmd has higher priority and programs in
        // there will be called in favor of those in \bin.
        EnvPath[i]:=AppDir+'\cmd';

        if RdbPath[GP_CmdTools].Checked then begin
            SetArrayLength(EnvPath,i+3);
            EnvPath[i+1]:=AppDir+'\{#MINGW_BITNESS}\bin';
            EnvPath[i+2]:=AppDir+'\usr\bin';
        end;
    end;

    // Set the current user's PATH directories.
    if not SetEnvStrings('PATH',EnvPath,True,IsAdminLoggedOn,True) then
        LogError('Line {#__LINE__}: Unable to set the PATH environment variable.');

Giving, I hope, sufficient info to modify my path locally.

[dscho]

Giving, I hope, sufficient info to modify my path locally.

Yes. Or looking at what I said at #2349 (comment)

[PhilipOakley]

I think we overlapped by a minute 😉

My main aim was to capture the issue for others so that those who'd used the same install option as me didn't think the vs/master was a failure.

[PhilipOakley]

I've locally added a user feedback 'fix'. Just need to decide where to create the PR patch.

phili@Philip-Win10 MINGW64 /c/git-sdk-64/usr/src/VisualStudioGit (vs/master)
$ git diff
diff --git a/compat/vcbuild/vcpkg_install.bat b/compat/vcbuild/vcpkg_install.bat
index ebd0bad242a..8f89cf6977b 100644
--- a/compat/vcbuild/vcpkg_install.bat
+++ b/compat/vcbuild/vcpkg_install.bat
@@ -36,6 +36,13 @@ REM ================================================================

        dir vcpkg\vcpkg.exe >nul 2>nul && GOTO :install_libraries

+       git.exe version 2>nul
+       IF ERRORLEVEL 1 (
+       echo "***"
+       echo "Git not found. Please adjust you CMD path or Git install option."
+       echo "***"
+       EXIT /B 1 )
+
        echo Fetching vcpkg in %cwd%vcpkg
        git.exe clone https://github.com/Microsoft/vcpkg vcpkg
        IF ERRORLEVEL 1 ( EXIT /B 1 )

phili@Philip-Win10 MINGW64 /c/git-sdk-64/usr/src/VisualStudioGit (vs/master)

I'll probably also adjust https://github.com/git-for-windows/build-extra/blob/master/installer/install.iss#L1666 to make the '(recommended)' into a red standout. git-for-windows/build-extra#258

[PhilipOakley]

I let the build run, and a few errors were showing roughly along the lines that zlib, ssl etc weren't right.

Like a fool I cleared the window to start a fresh build, and the initial messages are

1>------ Build started: Project: libgit, Configuration: Debug x64 ------
1>Installing third-party libraries...
1>    Installing package zlib...
1>The following packages will be built and installed:
1>    zlib[core]:x64-windows
1>Starting package 1/1: zlib:x64-windows
1>Building package zlib[core]:x64-windows...

looking good so far.
Note that this is from a worktree that is to the side of the regular Git main worktree.

1>------ Build started: Project: libgit, Configuration: Debug x64 ------
1>Installing third-party libraries...
1>    Installing package zlib...
1>The following packages will be built and installed:
1>    zlib[core]:x64-windows
1>Starting package 1/1: zlib:x64-windows
1>Building package zlib[core]:x64-windows...
1>A suitable version of powershell-core was not found (required v6.2.1). Downloading portable powershell-core v6.2.1...
1>Downloading powershell-core...
1>  https://github.com/PowerShell/PowerShell/releases/download/v6.2.1/PowerShell-6.2.1-win-x86.zip -> C:\git-sdk-64\usr\src\VisualStudioGit\compat\vcbuild\vcpkg\downloads\PowerShell-6.2.1-win-x86.zip
1>WinHttpQueryDataAvailable() failed: 12002
1>C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\V140\Microsoft.CppCommon.targets(123,5): error MSB3073: The command "call "..\compat\vcbuild\vcpkg_install.bat"
1>C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\V140\Microsoft.CppCommon.targets(123,5): error MSB3073: :VCEnd" exited with code 1.
1>Done building project "libgit.vcxproj" -- FAILED.
2>------ Build started: Project: xdiff_lib, Configuration: Debug x64 ------
2>xutils.c
2>c:\git-sdk-64\usr\src\visualstudiogit\compat/mingw.h(178): fatal error C1083: Cannot open include file: 'openssl/ssl.h': No such file or directory
2>xprepare.c
...

Hmm, still errors. More to do.

The (again) build Error List window showed

Severity	Code	Description	Project	File	Line	Suppression State
Error (active)	E1696	cannot open source file "openssl/x509v3.h"	git	C:\git-sdk-64\usr\src\VisualStudioGit\git-compat-util.h	495	
Error (active)	E1696	cannot open source file "zlib.h"	git	C:\git-sdk-64\usr\src\VisualStudioGit\cache.h	21	
Error (active)	E1696	cannot open source file "openssl/ssl.h"	git	C:\git-sdk-64\usr\src\VisualStudioGit\compat\mingw.h	178	
Error (active)	E1696	cannot open source file "iconv.h"	git	C:\git-sdk-64\usr\src\VisualStudioGit\git-compat-util.h	288	
Error (active)	E1696	cannot open source file "openssl/ssl.h"	git	C:\git-sdk-64\usr\src\VisualStudioGit\git-compat-util.h	299	
Error (active)	E1696	cannot open source file "openssl/err.h"	git	C:\git-sdk-64\usr\src\VisualStudioGit\git-compat-util.h	300	
Error (active)	E1696	cannot open source file "openssl/evp.h"	git	C:\git-sdk-64\usr\src\VisualStudioGit\git-compat-util.h	492	
Error (active)	E1696	cannot open source file "openssl/hmac.h"	git	C:\git-sdk-64\usr\src\VisualStudioGit\git-compat-util.h	493	

[PhilipOakley]

So, it looks like the vcpkg.exe is only downloading part of the required files before giving up. (word wrapped for clarity)

1>Building package zlib[core]:x64-windows...
1>A suitable version of powershell-core was not found (required v6.2.1).
  Downloading portable powershell-core v6.2.1...
1>Downloading powershell-core...
1>  https://github.com/PowerShell/PowerShell/releases/download
/v6.2.1/PowerShell-6.2.1-win-x86.zip
 -> C:\git-sdk-64\usr\src\VisualStudioGit
\compat\vcbuild\vcpkg\downloads\PowerShell-6.2.1-win-x86.zip
1>WinHttpQueryDataAvailable() failed: 12002

Explorer shows
C:\git-sdk-64\usr\src\VisualStudioGit\compat\vcbuild\vcpkg\downloads\PowerShell-6.2.1-win-x86.zip.part date/time 41,004KB (note this is a .part file)

So why the WinHttpQueryDataAvailable() failed: 12002 ?

Let's delete that .part file and try again, while googling that error!

[PhilipOakley]

well googling "vcpkg winhttpquerydataavailable() failed 12002" top 3 show that vcpkg can be temperamental across slow, intermittent or proxy connections, so simply retry (perhaps removing that .part file in case it's corrupt) and go again.

A bit further this time.

1>Building package openssl-windows[core]:x64-windows...
1>-- Downloading http://strawberryperl.com/download/5.30.0.1/strawberry-perl-5.30.0.1-32bit.zip...
1>-- Downloading https://www.openssl.org/source/openssl-1.0.2s.tar.gz...
1>-- Extracting source C:/git-sdk-64/usr/src/VisualStudioGit/compat/vcbuild/vcpkg/downloads/openssl-1.0.2s.tar.gz
1>-- Applying patch ConfigureIncludeQuotesFix.patch
1>-- Applying patch STRINGIFYPatch.patch
1>-- Applying patch EnableWinARM32.patch
1>-- Applying patch EmbedSymbolsInStaticLibsZ7.patch
1>-- Applying patch EnableWinARM64.patch
1>-- Using source at C:/git-sdk-64/usr/src/VisualStudioGit/compat/vcbuild/vcpkg/buildtrees/openssl-windows/src/openssl-1-c41872ef97
1>-- Downloading http://www.nasm.us/pub/nasm/releasebuilds/2.14.02/win32/nasm-2.14.02-win32.zip...
1>CMake Error at scripts/cmake/vcpkg_download_distfile.cmake:162 (file):
1>EXEC : file DOWNLOAD error : cannot create directory
1>  'C:/git-sdk-64/usr/src/VisualStudioGit/compat/vcbuild/vcpkg/downloads/temp'
1>  - Specify file by full path name and verify that you have directory
1>  creation and file write privileges.
1>Call Stack (most recent call first):
1>  scripts/cmake/vcpkg_find_acquire_program.cmake:294 (vcpkg_download_distfile)
1>  ports/openssl-windows/portfile.cmake:37 (vcpkg_find_acquire_program)
1>  scripts/ports.cmake:94 (include)
1>
1>
1>EXEC : error : Building package openssl-windows:x64-windows failed with: BUILD_FAILED
1>Please ensure you're using the latest portfiles with `.\vcpkg update`, then
1>submit an issue at https://github.com/Microsoft/vcpkg/issues including:
1>  Package: openssl-windows:x64-windows
1>  Vcpkg version: 2019.09.12-nohash
1>
1>Additionally, attach any relevant sections from the log files above.

Maybe just go again and hope (and review those errors?)

[PhilipOakley]

got past the vcpkg issues. I'm adding a comment

@@ -73,6 +80,11 @@ REM ================================================================
 :sub__install_one
        echo     Installing package %1...

+REM vcpkg may not be reliable on slow, intermittent or proxy connections
+REM e.g. https://social.msdn.microsoft.com/Forums/windowsdesktop/en-US/4a8f7be5-5e15-4213-a7bb-ddf424a954e6/winhttpsendrequest-ends-with-12002-errorhttptimeout-after-21-seconds-no-matter-what-timeout?forum=windowssdk
+REM which explains the hidden 21 second timeout
+REM (last post by Dave : Microsoft - Windows Networking team)
+
        .\vcpkg.exe install %1:%arch%
        IF ERRORLEVEL 1 ( EXIT /B 1 )

to help folk coming later.

However there was still one (probably non-)issue in the compile with one of the .pdb files (line wrapped)

6>C:\git-sdk-64\usr\src\VisualStudioGit\git-daemon\git-daemon.vcxproj(134,5):
 warning MSB3026:
 Could not copy "..\compat\vcbuild\vcpkg\installed\x64-windows\bin\libssh2.pdb"
 to "..\libssh2.pdb". Beginning retry 1 in 1000ms.
 The process cannot access the file '..\libssh2.pdb' because it is being used by another process.

Not sure where the race is. But we have progress.