Skip to content

Commit 4ee8d34

Browse files
jeffhostetlerdscho
jeffhostetler
authored and
dscho
committed
vcpkg: get MSVC dependencies with vcpkg rather than nuget
Dependencies such as cURL and OpenSSL are necessary to build and run Git. Previously, we obtained those dependencies by fetching NuGet packages. However, it is notoriously hard to keep NuGet packages of C/C++ libraries up-to-date, as the toolsets for different Visual Studio versions are different, and the NuGet packages would have to ship them all. That is the reason why the NuGet packages we use are quite old, and even insecure in the case of cURL and OpenSSL (the versions contain known security flaws that have been addressed by later versions for which no NuGet packages are available). The better way to handle this situation is to use the vcpkg system: https://github.com/Microsoft/vcpkg The idea is that a single Git repository contains enough supporting files to build up-to-date versions of a large number of Open Source libraries on demand, including cURL and OpenSSL. We integrate this system via four new .bat files to 1) initialize the vcpkg system, 2) build the packages, 4) set up Git's Makefile system to find the build artifacts, and 3) copy the artifacts into the top-level directory We now also completely convert the pathname exported in the %msvc_bin_dir_msys% variable to MSYS format with forward slashes rather than a mixture of forward and back slashes. This solves an obscure problem observed by some developers: [...] http-push.c CC remote-curl.o remote-curl.c * new script parameters GEN git-instaweb sed: -e expression #7, char 155: invalid reference \2 on `s' command's RHS make: *** [Makefile:2023: git-instaweb] Error 1 Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
1 parent 85fda73 commit 4ee8d34

File tree

10 files changed

+215
-25
lines changed

10 files changed

+215
-25
lines changed

‎.gitignore

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,6 @@
11
/GIT-BUILD-OPTIONS
22
/GIT-CFLAGS
33
/GIT-LDFLAGS
4-
/GIT-MSVC-GEN
54
/GIT-PREFIX
65
/GIT-PERL-DEFINES
76
/GIT-PERL-HEADER

‎Makefile

Lines changed: 7 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -2602,9 +2602,6 @@ ifdef GIT_INTEROP_MAKE_OPTS
26022602
endif
26032603
ifdef TEST_GIT_INDEX_VERSION
26042604
@echo TEST_GIT_INDEX_VERSION=\''$(subst ','\'',$(subst ','\'',$(TEST_GIT_INDEX_VERSION)))'\' >>$@+
2605-
endif
2606-
ifdef MSVC_DEPS
2607-
@echo MSVC_DEPS=\''$(subst ','\'',$(subst ','\'',$(MSVC_DEPS)))'\' >>$@+
26082605
endif
26092606
@if cmp $@+ $@ >/dev/null 2>&1; then $(RM) $@+; else mv $@+ $@; fi
26102607

@@ -2754,8 +2751,6 @@ install: all
27542751
$(INSTALL) -m 644 $(SCRIPT_LIB) '$(DESTDIR_SQ)$(gitexec_instdir_SQ)'
27552752
$(INSTALL) $(install_bindir_programs) '$(DESTDIR_SQ)$(bindir_SQ)'
27562753
ifdef MSVC
2757-
$(INSTALL) compat/vcbuild/GEN.DEPS/bin/*.dll '$(DESTDIR_SQ)$(bindir_SQ)'
2758-
$(INSTALL) compat/vcbuild/GEN.DEPS/bin/*.pdb '$(DESTDIR_SQ)$(bindir_SQ)'
27592754
# We DO NOT install the individual foo.o.pdb files because they
27602755
# have already been rolled up into the exe's pdb file.
27612756
# We DO NOT have pdb files for the builtin commands (like git-status.exe)
@@ -2978,9 +2973,15 @@ endif
29782973
ifdef MSVC
29792974
$(RM) $(patsubst %.o,%.o.pdb,$(OBJECTS))
29802975
$(RM) $(patsubst %.exe,%.pdb,$(OTHER_PROGRAMS))
2976+
$(RM) $(patsubst %.exe,%.iobj,$(OTHER_PROGRAMS))
2977+
$(RM) $(patsubst %.exe,%.ipdb,$(OTHER_PROGRAMS))
29812978
$(RM) $(patsubst %.exe,%.pdb,$(PROGRAMS))
2979+
$(RM) $(patsubst %.exe,%.iobj,$(PROGRAMS))
2980+
$(RM) $(patsubst %.exe,%.ipdb,$(PROGRAMS))
29822981
$(RM) $(patsubst %.exe,%.pdb,$(TEST_PROGRAMS))
2983-
$(RM) GIT-MSVC-GEN
2982+
$(RM) $(patsubst %.exe,%.iobj,$(TEST_PROGRAMS))
2983+
$(RM) $(patsubst %.exe,%.ipdb,$(TEST_PROGRAMS))
2984+
$(RM) compat/vcbuild/MSVC-DEFS-GEN
29842985
endif
29852986

29862987
.PHONY: all install profile-clean cocciclean clean strip

‎compat/vcbuild/.gitignore

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1 +1,3 @@
1-
GEN.*
1+
/vcpkg/
2+
/MSVC-DEFS-GEN
3+
/VCPKG-DEFS

‎compat/vcbuild/README

Lines changed: 51 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,54 @@
1+
The Steps to Build Git with VS2015 or VS2017 from the command line.
2+
3+
1. Install the "vcpkg" open source package manager and build essential
4+
third-party libaries. The steps for this have been captured in a
5+
set of convenience scripts. These can be run from a stock Command
6+
Prompt or from an SDK bash window:
7+
8+
$ cd <repo_root>
9+
$ ./compat/vcbuild/vcpkg_install.bat
10+
11+
The vcpkg tools and all of the third-party sources will be installed
12+
in this folder:
13+
<repo_root>/compat/vcbuild/vcpkg/
14+
15+
A file will be created with a set of Makefile macros pointing to a
16+
unified "include", "lib", and "bin" directory (release and debug) for
17+
all of the required packages. This file will be included by the main
18+
Makefile:
19+
<repo_root>/compat/vcbuild/MSVC-DEFS-GEN
20+
21+
2. OPTIONALLY copy the third-party *.dll and *.pdb files into the repo
22+
root to make it easier to run and debug git.exe without having to
23+
manipulate your PATH. This is especially true for debug sessions in
24+
Visual Studio.
25+
26+
Use ONE of the following forms which should match how you want to
27+
compile git.exe.
28+
29+
$ ./compat/vcbuild/vcpkg_copy_packages.bat debug
30+
$ ./compat/vcbuild/vcpkg_copy_packages.bat release
31+
32+
3. Build git using MSVC from an SDK bash window using one of the
33+
following commands:
34+
35+
$ make MSVC=1
36+
$ make MSVC=1 DEBUG=1
37+
38+
================================================================
39+
40+
Alternatively, run `make MSVC=1 vcxproj` and then load the generated
41+
git.sln in Visual Studio. The initial build will install the vcpkg
42+
system and build the dependencies automatically. This will take a while.
43+
44+
Note that this will automatically add and commit the generated
45+
.sln and .vcxproj files to the repo. You may want to drop this
46+
commit before submitting a Pull Request....
47+
48+
Or maybe we should put the .sln/.vcxproj files in the .gitignores
49+
and not do this. I'm not sure.
50+
51+
================================================================
152
The Steps of Build Git with VS2008
253

354
1. You need the build environment, which contains the Git dependencies

‎compat/vcbuild/find_vs_env.bat

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -25,7 +25,7 @@ REM
2525
REM The output of this script should be written to a make "include
2626
REM file" and referenced by the top-level Makefile.
2727
REM
28-
REM See "config.mak.uname" (look for GIT-MSVC-GEN).
28+
REM See "config.mak.uname" (look for compat/vcbuild/MSVC-DEFS-GEN).
2929
REM ================================================================
3030
REM The provided command prompts are custom to each VS release and
3131
REM filled with lots of internal knowledge (such as Registry settings);
@@ -154,7 +154,9 @@ REM ================================================================
154154
REM Include DOS-style and BASH-style path for bin dir.
155155

156156
echo msvc_bin_dir=%msvc_bin_dir%
157-
echo msvc_bin_dir_msys=%msvc_bin_dir:C:=/C%
157+
SET X1=%msvc_bin_dir:C:=/C%
158+
SET X2=%X1:\=/%
159+
echo msvc_bin_dir_msys=%X2%
158160

159161
echo msvc_includes=%msvc_includes%
160162
echo msvc_libs=%msvc_libs%

‎compat/vcbuild/scripts/clink.pl

Lines changed: 13 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -14,8 +14,15 @@
1414
my @cflags = ();
1515
my @lflags = ();
1616
my $is_linking = 0;
17+
my $is_debug = 0;
1718
while (@ARGV) {
1819
my $arg = shift @ARGV;
20+
if ("$arg" eq "-DDEBUG") {
21+
# Some vcpkg-based libraries have different names for release
22+
# and debug versions. This hack assumes that -DDEBUG comes
23+
# before any "-l*" flags.
24+
$is_debug = 1;
25+
}
1926
if ("$arg" =~ /^-[DIMGOZ]/) {
2027
push(@cflags, $arg);
2128
} elsif ("$arg" eq "-o") {
@@ -30,17 +37,21 @@
3037
push(@args, "-Fd$file_out.pdb");
3138
}
3239
} elsif ("$arg" eq "-lz") {
40+
if ($is_debug) {
41+
push(@args, "zlibd.lib");
42+
} else{
3343
push(@args, "zlib.lib");
44+
}
3445
} elsif ("$arg" eq "-liconv") {
35-
push(@args, "iconv.lib");
46+
push(@args, "libiconv.lib");
3647
} elsif ("$arg" eq "-lcrypto") {
3748
push(@args, "libeay32.lib");
3849
} elsif ("$arg" eq "-lssl") {
3950
push(@args, "ssleay32.lib");
4051
} elsif ("$arg" eq "-lcurl") {
4152
push(@args, "libcurl.lib");
4253
} elsif ("$arg" eq "-lexpat") {
43-
push(@args, "libexpat.lib");
54+
push(@args, "expat.lib");
4455
} elsif ("$arg" =~ /^-L/ && "$arg" ne "-LTCG") {
4556
$arg =~ s/^-L/-LIBPATH:/;
4657
push(@lflags, $arg);

‎compat/vcbuild/vcpkg_copy_dlls.bat

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
@ECHO OFF
2+
REM ================================================================
3+
REM This script is an optional step. It copies the *.dll and *.pdb
4+
REM files (created by vcpkg_install.bat) into the top-level directory
5+
REM of the repo so that you can type "./git.exe" and find them without
6+
REM having to fixup your PATH.
7+
REM
8+
REM NOTE: Because the names of some DLL files change between DEBUG and
9+
REM NOTE: RELEASE builds when built using "vcpkg.exe", you will need
10+
REM NOTE: to copy up the corresponding version.
11+
REM ================================================================
12+
13+
SETLOCAL EnableDelayedExpansion
14+
15+
@FOR /F "delims=" %%D IN ("%~dp0") DO @SET cwd=%%~fD
16+
cd %cwd%
17+
18+
SET arch=x64-windows
19+
SET inst=%cwd%vcpkg\installed\%arch%
20+
21+
IF [%1]==[release] (
22+
echo Copying RELEASE mode DLLs to repo root...
23+
) ELSE IF [%1]==[debug] (
24+
SET inst=%inst%\debug
25+
echo Copying DEBUG mode DLLs to repo root...
26+
) ELSE (
27+
echo ERROR: Invalid argument.
28+
echo Usage: %~0 release
29+
echo Usage: %~0 debug
30+
EXIT /B 1
31+
)
32+
33+
xcopy /e/s/v/y %inst%\bin\*.dll ..\..\
34+
xcopy /e/s/v/y %inst%\bin\*.pdb ..\..\
35+
36+
xcopy /e/s/v/y %inst%\bin\*.dll ..\..\t\helper\
37+
xcopy /e/s/v/y %inst%\bin\*.pdb ..\..\t\helper\
38+
39+
EXIT /B 0

‎compat/vcbuild/vcpkg_install.bat

Lines changed: 81 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,81 @@
1+
@ECHO OFF
2+
REM ================================================================
3+
REM This script installs the "vcpkg" source package manager and uses
4+
REM it to build the third-party libraries that git requires when it
5+
REM is built using MSVC.
6+
REM
7+
REM [1] Install VCPKG.
8+
REM [a] Create <root>/compat/vcbuild/vcpkg/
9+
REM [b] Download "vcpkg".
10+
REM [c] Compile using the currently installed version of VS.
11+
REM [d] Create <root>/compat/vcbuild/vcpkg/vcpkg.exe
12+
REM
13+
REM [2] Install third-party libraries.
14+
REM [a] Download each (which may also install CMAKE).
15+
REM [b] Compile in RELEASE mode and install in:
16+
REM vcpkg/installed/<arch>/{bin,lib}
17+
REM [c] Compile in DEBUG mode and install in:
18+
REM vcpkg/installed/<arch>/debug/{bin,lib}
19+
REM [d] Install headers in:
20+
REM vcpkg/installed/<arch>/include
21+
REM
22+
REM [3] Create a set of MAKE definitions for the top-level
23+
REM Makefile to allow "make MSVC=1" to find the above
24+
REM third-party libraries.
25+
REM [a] Write vcpkg/VCPGK-DEFS
26+
REM
27+
REM https://blogs.msdn.microsoft.com/vcblog/2016/09/19/vcpkg-a-tool-to-acquire-and-build-c-open-source-libraries-on-windows/
28+
REM https://github.com/Microsoft/vcpkg
29+
REM https://vcpkg.readthedocs.io/en/latest/
30+
REM ================================================================
31+
32+
SETLOCAL EnableDelayedExpansion
33+
34+
@FOR /F "delims=" %%D IN ("%~dp0") DO @SET cwd=%%~fD
35+
cd %cwd%
36+
37+
dir vcpkg\vcpkg.exe >nul 2>nul && GOTO :install_libraries
38+
39+
echo Fetching vcpkg in %cwd%vcpkg
40+
git.exe clone https://github.com/Microsoft/vcpkg vcpkg
41+
IF ERRORLEVEL 1 ( EXIT /B 1 )
42+
43+
cd vcpkg
44+
echo Building vcpkg
45+
powershell -exec bypass scripts\bootstrap.ps1
46+
IF ERRORLEVEL 1 ( EXIT /B 1 )
47+
48+
echo Successfully installed %cwd%vcpkg\vcpkg.exe
49+
50+
:install_libraries
51+
SET arch=x64-windows
52+
53+
echo Installing third-party libraries...
54+
FOR %%i IN (zlib expat libiconv openssl libssh2 curl) DO (
55+
cd %cwd%vcpkg
56+
SET p="packages\%%i_%arch%"
57+
IF NOT EXIST "%p%" CALL :sub__install_one %%i
58+
IF ERRORLEVEL 1 ( EXIT /B 1 )
59+
)
60+
61+
:install_defines
62+
cd %cwd%
63+
SET inst=%cwd%vcpkg\installed\%arch%
64+
65+
echo vcpkg_inc=-I"%inst%\include">VCPKG-DEFS
66+
echo vcpkg_rel_lib=-L"%inst%\lib">>VCPKG-DEFS
67+
echo vcpkg_rel_bin="%inst%\bin">>VCPKG-DEFS
68+
echo vcpkg_dbg_lib=-L"%inst%\debug\lib">>VCPKG-DEFS
69+
echo vcpkg_dbg_bin="%inst%\debug\bin">>VCPKG-DEFS
70+
71+
EXIT /B 0
72+
73+
74+
:sub__install_one
75+
echo Installing package %1...
76+
77+
.\vcpkg.exe install %1:%arch%
78+
IF ERRORLEVEL 1 ( EXIT /B 1 )
79+
80+
echo Finished %1
81+
goto :EOF

‎config.mak.uname

Lines changed: 17 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -18,9 +18,16 @@ ifdef MSVC
1818

1919
# Generate and include makefile variables that point to the
2020
# currently installed set of MSVC command line tools.
21-
GIT-MSVC-GEN: ./compat/vcbuild/find_vs_env.bat
22-
@./compat/vcbuild/find_vs_env.bat | sed 's|\\|/|g' >GIT-MSVC-GEN
23-
-include GIT-MSVC-GEN
21+
compat/vcbuild/MSVC-DEFS-GEN: compat/vcbuild/find_vs_env.bat
22+
@"$<" | tr '\\' / >"$@"
23+
include compat/vcbuild/MSVC-DEFS-GEN
24+
25+
# See if vcpkg and the vcpkg-build versions of the third-party
26+
# libraries that we use are installed. We include the result
27+
# to get $(vcpkg_*) variables defined for the Makefile.
28+
compat/vcbuild/VCPKG-DEFS: compat/vcbuild/vcpkg_install.bat
29+
@"$<"
30+
include compat/vcbuild/VCPKG-DEFS
2431
endif
2532

2633
# We choose to avoid "if .. else if .. else .. endif endif"
@@ -427,13 +434,13 @@ ifeq ($(uname_S),Windows)
427434
EXTLIBS = user32.lib advapi32.lib shell32.lib wininet.lib ws2_32.lib invalidcontinue.obj kernel32.lib ntdll.lib
428435
PTHREAD_LIBS =
429436
lib =
430-
# Path to the unpacked third-party libraries
431-
MSVC_DEPS = compat/vcbuild/GEN.DEPS
432-
BASIC_CFLAGS += \
433-
-I$(MSVC_DEPS)/include -I$(MSVC_DEPS)/include/expat -I$(MSVC_DEPS)/include/zlib \
434-
-L$(MSVC_DEPS)/lib \
435-
$(sdk_includes) $(sdk_libs) \
436-
$(msvc_includes) $(msvc_libs)
437+
BASIC_CFLAGS += $(vcpkg_inc) $(sdk_includes) $(msvc_includes)
438+
ifndef DEBUG
439+
BASIC_CFLAGS += $(vcpkg_rel_lib)
440+
else
441+
BASIC_CFLAGS += $(vcpkg_dbg_lib)
442+
endif
443+
BASIC_CFLAGS += $(sdk_libs) $(msvc_libs)
437444

438445
# Optionally enable memory leak reporting.
439446
# BASIC_CLFAGS += -DUSE_MSVC_CRTDBG

‎t/test-lib.sh

Lines changed: 0 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -57,9 +57,6 @@ fi
5757
. "$GIT_BUILD_DIR"/GIT-BUILD-OPTIONS
5858
export PERL_PATH SHELL_PATH
5959

60-
test -z "$MSVC_DEPS" ||
61-
PATH="$GIT_BUILD_DIR/$MSVC_DEPS/bin:$PATH"
62-
6360
################################################################
6461
# It appears that people try to run tests without building...
6562
test -n "$GIT_TEST_INSTALLED" || "$GIT_BUILD_DIR/git$X" >/dev/null ||

0 commit comments

Comments
 (0)