Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

git-dependency testcase is broken on Windows #2443

Closed
shoo opened this issue Aug 28, 2022 · 5 comments
Closed

git-dependency testcase is broken on Windows #2443

shoo opened this issue Aug 28, 2022 · 5 comments

Comments

@shoo
Copy link
Contributor

shoo commented Aug 28, 2022

System information

  • dub version: dub (master branch / commit: a8aa234)
  • OS Platform and distribution: Windows 11
  • compiler version dmd-2.100.0 / ldc-1.30.0

Bug Description

test/git-dependency is not work on Windows after #2386

This problem occurs only on Windows. The contents of test/git-dependency test case show that it can be built on Windows.
It is also a regression, as it builds successfully on past versions.
The point at which the problem occurred was tracked down by git bisect.

How to reproduce?

git checkout a8aa2341be8434b16630aba03958f70565ff3ad6
dub run -- build --root=test/git-dependency

Expected Behavior

Successful builds, even on Windows

Logs

Log
R:\dub> dub run -- build --root .\test\git-dependency -v
Performing "debug" build using P:\app\dmd\bin64\dmd.exe for x86_64.
dub ~master: building configuration "application"...
Serializing composite type Flags!(BuildRequirement) which has no serializable fields
Serializing composite type Flags!(BuildOption) which has no serializable fields
Linking...
Running bin/dub.exe build --root .\test\git-dependency -v
Using dub registry url 'https://code.dlang.org/'
Refreshing local packages (refresh existing: true)...
......
......
......
Looking for local package map at R:\dub\test\git-dependency\.dub\packages\local-packages.json
  Version selection for dependency gitcompatibledubpackage (gitcompatibledubpackage) of git-dependency is missing.
  Found dependency gitcompatibledubpackage git+https://github.com/dlang-community/gitcompatibledubpackage.git
Generating using build
Configuring dependent git-dependency, deps:"gitcompatibledubpackage"
  Configuring dependent gitcompatibledubpackage, deps:
    Starting Performing "debug" build using dmd for x86_64.
Target '......\AppData\Local\dub\packages\gitcompatibledubpackage-ccb31bf6a655437176ec02e04c2305a8c7c90d67\gitcompatibledubpackage\.dub\build\lib-debug-windows-x86_64-dmd_v2.100.1-987A8C41F524B24BEFD436F70852ACB9ED35FAEA102BE102629C2BEA03A44814\gitcompatibledubpackage.lib' doesn't exist, need rebuild.
             Build directory ......\AppData\Local\dub\packages\gitcompatibledubpackage-ccb31bf6a655437176ec02e04c2305a8c7c90d67\gitcompatibledubpackage\.dub\build\lib-debug-windows-x86_64-dmd_v2.100.1-987A8C41F524B24BEFD436F70852ACB9ED35FAEA102BE102629C2BEA03A44814\ is not writable. Falling back to direct build in the system's temp folder.
    Building gitcompatibledubpackage 1.0.4+commit.2.gccb31bf [lib]
dmd -m64 -lib -of......\AppData\Local\dub\packages\gitcompatibledubpackage-ccb31bf6a655437176ec02e04c2305a8c7c90d67\gitcompatibledubpackage\gitcompatibledubpackage.lib -debug -g -w -version=Have_gitcompatibledubpackage -I......\AppData\Local\dub\packages\gitcompatibledubpackage-ccb31bf6a655437176ec02e04c2305a8c7c90d67 ......\AppData\Local\dub\packages\gitcompatibledubpackage-ccb31bf6a655437176ec02e04c2305a8c7c90d67\gitcompatibledubpackage\subdir\file.d -vcolumns
Target 'R:\dub\test\git-dependency\.dub\build\application-debug-windows-x86_64-dmd_v2.100.1-5C3FC3BA134840FB42131EF1DCE7C916264FCEB8161FDA0557EA48231B93C420\git-dependency.exe' doesn't exist, need rebuild.
    Building git-dependency ~master: building configuration [application]
dmd -m64 -c -oftest\git-dependency\.dub\build\application-debug-windows-x86_64-dmd_v2.100.1-5C3FC3BA134840FB42131EF1DCE7C916264FCEB8161FDA0557EA48231B93C420\git-dependency.obj -debug -g -w -version=Have_git_dependency -version=Have_gitcompatibledubpackage -Itest\git-dependency\src -I......\AppData\Local\dub\packages\gitcompatibledubpackage-ccb31bf6a655437176ec02e04c2305a8c7c90d67 test\git-dependency\src\app.d -vcolumns
     Linking git-dependency
dmd -oftest\git-dependency\.dub\build\application-debug-windows-x86_64-dmd_v2.100.1-5C3FC3BA134840FB42131EF1DCE7C916264FCEB8161FDA0557EA48231B93C420\git-dependency.exe test\git-dependency\.dub\build\application-debug-windows-x86_64-dmd_v2.100.1-5C3FC3BA134840FB42131EF1DCE7C916264FCEB8161FDA0557EA48231B93C420\git-dependency.obj -m64 -g
git-dependency.obj : error LNK2019: 未解決の外部シンボル _D23gitcompatibledubpackage6subdir4file19hasTheWorldExplodedFZb が関数 _Dmain で参照されました。
test\git-dependency\.dub\build\application-debug-windows-x86_64-dmd_v2.100.1-5C3FC3BA134840FB42131EF1DCE7C916264FCEB8161FDA0557EA48231B93C420\git-dependency.exe : fatal error LNK1120: 1 件の未解決の外部参照
Error: linker exited with status 1120
FAIL test\git-dependency\.dub\build\application-debug-windows-x86_64-dmd_v2.100.1-5C3FC3BA134840FB42131EF1DCE7C916264FCEB8161FDA0557EA48231B93C420\ git-dependency executable
Error dmd failed with exit code 1.
Program exited with code 2
Log before #2386
R:\dub> git checkout 71ae566bc2630b73e386def15eb2f5bbb494ade1 -q
R:\dub> dub run -- build --root .\test\git-dependency -v
Performing "debug" build using P:\app\dmd\bin64\dmd.exe for x86_64.
dub 1.29.1+commit.215.g71ae566b: building configuration "application"...
Serializing composite type Flags!(BuildRequirement) which has no serializable fields
Serializing composite type Flags!(BuildOption) which has no serializable fields
Linking...
Running bin/dub.exe build --root .\test\git-dependency -v
Using dub registry url 'https://code.dlang.org/'
Refreshing local packages (refresh existing: true)...
......
......
......
Looking for local package map at R:\dub\test\git-dependency\.dub\packages\local-packages.json
  Version selection for dependency gitcompatibledubpackage (gitcompatibledubpackage) of git-dependency is missing.
  Found dependency gitcompatibledubpackage git+https://github.com/dlang-community/gitcompatibledubpackage.git
Generating using build
Configuring dependent git-dependency, deps:"gitcompatibledubpackage"
  Configuring dependent gitcompatibledubpackage, deps:
    Starting Performing "debug" build using dmd for x86_64.
  Up-to-date gitcompatibledubpackage 1.0.4+commit.2.gccb31bf: target for configuration [lib] is up to date.
Using existing build in ......\AppData\Local\dub\packages\gitcompatibledubpackage-ccb31bf6a655437176ec02e04c2305a8c7c90d67\gitcompatibledubpackage\.dub\build\lib-debug-windows-x86_64-dmd_v2.100.1-3B8E4AEDEEC3AEE060DBE78BC72E6118\.
Copying target from ......\AppData\Local\dub\packages\gitcompatibledubpackage-ccb31bf6a655437176ec02e04c2305a8c7c90d67\gitcompatibledubpackage\.dub\build\lib-debug-windows-x86_64-dmd_v2.100.1-3B8E4AEDEEC3AEE060DBE78BC72E6118\gitcompatibledubpackage.lib to ......\AppData\Local\dub\packages\gitcompatibledubpackage-ccb31bf6a655437176ec02e04c2305a8c7c90d67\gitcompatibledubpackage
Target 'R:\dub\test\git-dependency\.dub\build\application-debug-windows-x86_64-dmd_v2.100.1-D791E20AFF2E02A675F577B266B98B87\git-dependency.exe' doesn't exist, need rebuild.
    Building git-dependency ~master: building configuration [application]
dmd -m64 -c -oftest\git-dependency\.dub\build\application-debug-windows-x86_64-dmd_v2.100.1-D791E20AFF2E02A675F577B266B98B87\git-dependency.obj -debug -g -w -version=Have_git_dependency -version=Have_gitcompatibledubpackage -Itest\git-dependency\src -I......\AppData\Local\dub\packages\gitcompatibledubpackage-ccb31bf6a655437176ec02e04c2305a8c7c90d67 test\git-dependency\src\app.d -vcolumns
     Linking git-dependency
dmd -oftest\git-dependency\.dub\build\application-debug-windows-x86_64-dmd_v2.100.1-D791E20AFF2E02A675F577B266B98B87\git-dependency.exe test\git-dependency\.dub\build\application-debug-windows-x86_64-dmd_v2.100.1-D791E20AFF2E02A675F577B266B98B87\git-dependency.obj ......\AppData\Local\dub\packages\gitcompatibledubpackage-ccb31bf6a655437176ec02e04c2305a8c7c90d67\gitcompatibledubpackage\.dub\build\lib-debug-windows-x86_64-dmd_v2.100.1-3B8E4AEDEEC3AEE060DBE78BC72E6118\gitcompatibledubpackage.lib -m64 -g
Copying target from R:\dub\test\git-dependency\.dub\build\application-debug-windows-x86_64-dmd_v2.100.1-D791E20AFF2E02A675F577B266B98B87\git-dependency.exe to R:\dub\test\git-dependency
Copying target from R:\dub\test\git-dependency\.dub\build\application-debug-windows-x86_64-dmd_v2.100.1-D791E20AFF2E02A675F577B266B98B87\git-dependency.pdb to R:\dub\test\git-dependency
Copying target from ......\AppData\Local\dub\packages\gitcompatibledubpackage-ccb31bf6a655437176ec02e04c2305a8c7c90d67\gitcompatibledubpackage\.dub\build\lib-debug-windows-x86_64-dmd_v2.100.1-3B8E4AEDEEC3AEE060DBE78BC72E6118\gitcompatibledubpackage.lib to R:\dub\test\git-dependency
    Finished To force a rebuild of up-to-date targets, run again with --force
Log on linux(ubuntu)
~/dub# dub run -- clean --root ./test/git-dependency -v
Performing "debug" build using /dlang/dmd-2.100.0/linux/bin64/dmd for x86_64.
dub 1.29.1+commit.260.ga8aa2341: target for configuration "application" is up to date.
To force a rebuild of up-to-date targets, run again with --force.
Running bin/dub clean --root ./test/git-dependency -v
Using dub registry url 'https://code.dlang.org/'
Refreshing local packages (refresh existing: true)...
Looking for local package map at /var/lib/dub/packages/local-packages.json
Looking for local package map at /dlang/dub/packages/local-packages.json
Looking for local package map at /root/dub/test/git-dependency/.dub/packages/local-packages.json
Determined package version using GIT: gitcompatibledubpackage 1.0.4+commit.2.gccb31bf
Note: Failed to determine version of package git-dependency at .. Assuming ~master.
    Cleaning package at /root/dub/test/git-dependency
Note: Failed to determine version of package git-dependency at .. Assuming ~master.
root@b6290cb61961:~/dub# dub run -- build --root ./test/git-dependency -v
Performing "debug" build using /dlang/dmd-2.100.0/linux/bin64/dmd for x86_64.
dub 1.29.1+commit.260.ga8aa2341: target for configuration "application" is up to date.
To force a rebuild of up-to-date targets, run again with --force.
Running bin/dub build --root ./test/git-dependency -v
Using dub registry url 'https://code.dlang.org/'
Refreshing local packages (refresh existing: true)...
Looking for local package map at /var/lib/dub/packages/local-packages.json
Looking for local package map at /dlang/dub/packages/local-packages.json
Looking for local package map at /root/dub/test/git-dependency/.dub/packages/local-packages.json
Determined package version using GIT: gitcompatibledubpackage 1.0.4+commit.2.gccb31bf
Note: Failed to determine version of package git-dependency at .. Assuming ~master.
Refreshing local packages (refresh existing: false)...
Looking for local package map at /var/lib/dub/packages/local-packages.json
Looking for local package map at /dlang/dub/packages/local-packages.json
Looking for local package map at /root/dub/test/git-dependency/.dub/packages/local-packages.json
  Version selection for dependency gitcompatibledubpackage (gitcompatibledubpackage) of git-dependency is missing.
  Found dependency gitcompatibledubpackage git+https://github.com/dlang-community/gitcompatibledubpackage.git
Generating using build
Configuring dependent git-dependency, deps:"gitcompatibledubpackage"
  Configuring dependent gitcompatibledubpackage, deps:
    Starting Performing "debug" build using dmd for x86_64.
  Up-to-date gitcompatibledubpackage 1.0.4+commit.2.gccb31bf: target for configuration [lib] is up to date.
Using existing build in /dlang/dub/packages/gitcompatibledubpackage-ccb31bf6a655437176ec02e04c2305a8c7c90d67/gitcompatibledubpackage/.dub/build/lib-debug-linux.posix-x86_64-dmd_v2.100.0-2BD2AD6CE46F3CB081C4D45BADB78A189DE72B135C568890B14BAA3CBE87FE3D/.
Copying target from /dlang/dub/packages/gitcompatibledubpackage-ccb31bf6a655437176ec02e04c2305a8c7c90d67/gitcompatibledubpackage/.dub/build/lib-debug-linux.posix-x86_64-dmd_v2.100.0-2BD2AD6CE46F3CB081C4D45BADB78A189DE72B135C568890B14BAA3CBE87FE3D/libgitcompatibledubpackage.a to /dlang/dub/packages/gitcompatibledubpackage-ccb31bf6a655437176ec02e04c2305a8c7c90d67/gitcompatibledubpackage
Target '/root/dub/test/git-dependency/.dub/build/application-debug-linux.posix-x86_64-dmd_v2.100.0-46A626078465576416FA921BC9D36AA8C0C7D6654F16266056103E0FFE030FA4/git-dependency' doesn't exist, need rebuild.
    Building git-dependency ~master: building configuration [application]
dmd -c -oftest/git-dependency/.dub/build/application-debug-linux.posix-x86_64-dmd_v2.100.0-46A626078465576416FA921BC9D36AA8C0C7D6654F16266056103E0FFE030FA4/git-dependency.o -debug -g -w -version=Have_git_dependency -version=Have_gitcompatibledubpackage -Itest/git-dependency/src/ -I/dlang/dub/packages/gitcompatibledubpackage-ccb31bf6a655437176ec02e04c2305a8c7c90d67 test/git-dependency/src/app.d -vcolumns
     Linking git-dependency
dmd -oftest/git-dependency/.dub/build/application-debug-linux.posix-x86_64-dmd_v2.100.0-46A626078465576416FA921BC9D36AA8C0C7D6654F16266056103E0FFE030FA4/git-dependency test/git-dependency/.dub/build/application-debug-linux.posix-x86_64-dmd_v2.100.0-46A626078465576416FA921BC9D36AA8C0C7D6654F16266056103E0FFE030FA4/git-dependency.o /dlang/dub/packages/gitcompatibledubpackage-ccb31bf6a655437176ec02e04c2305a8c7c90d67/gitcompatibledubpackage/.dub/build/lib-debug-linux.posix-x86_64-dmd_v2.100.0-2BD2AD6CE46F3CB081C4D45BADB78A189DE72B135C568890B14BAA3CBE87FE3D/libgitcompatibledubpackage.a -L--no-as-needed -g
Copying target from /root/dub/test/git-dependency/.dub/build/application-debug-linux.posix-x86_64-dmd_v2.100.0-46A626078465576416FA921BC9D36AA8C0C7D6654F16266056103E0FFE030FA4/git-dependency to /root/dub/test/git-dependency
    Finished To force a rebuild of up-to-date targets, run again with --force

Logs have been partially processed.
@shoo
Copy link
Contributor Author

shoo commented Sep 2, 2022

Perhaps this happened because the length of the path exceeded MAX_PATH(260)?

@shoo
Copy link
Contributor Author

shoo commented Sep 4, 2022

I am suspicious of the following line:

dub/source/dub/generators/build.d

Line 242 in 32985ea

performDirectBuild(settings, buildsettings, pack, config, dependencyBinariesToCopy, target_path);

On Windows, if MAX_PATH(260) is exceeded, it is determined that writing is not possible and falling back to direct build in the system's temp folder.
The change in #2386 resulted in the creation of an intermediate folder with SHA256, which exceeded the MAX_PATH constraint, and the above process is now performed.
At this time, performDirectBuild is called and then the artifact must be stored in target_binary_path. However, target_path is specified in the function argument, and the artifact is not correctly propagated to the caller.

There are three solutions to this problem:

  1. Fix the fallback process to work properly as described above
  2. Make sure it builds correctly even if MAX_PATH is exceeded (this measure is actually difficult because the Visual Studio linker is unsupported)
  3. Make sure that MAX_PATH is not exceeded. In other words, the naming convention for intermediate folders must be changed.

In the short term, 1 may need to be fixed, but in the long term, I would prefer to fix 2 or 3.
Is my guess likely to be correct? And are there any other measures you can think of?

shoo added a commit to shoo/dub that referenced this issue Sep 4, 2022
@shoo
Fix dlang#2443 git-dependency testcase is broken on Windows
e8b4318
@shoo shoo mentioned this issue Sep 4, 2022
Closed
@Geod24
Copy link
Member

Geod24 commented Sep 5, 2022

Thanks for the great writeup!

Is my guess likely to be correct? And are there any other measures you can think of?

Likely, although I'm not sure why you mentioned the VS Linker is unsupported ? Also, 2 is definitely the way to go.

@shoo
Copy link
Contributor Author

shoo commented Sep 5, 2022

Likely, although I'm not sure why you mentioned the VS Linker is unsupported ?

The following commands cannot be linked cause of the path length:

R:\>"C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.22.27905\bin\HostX64\x64\link.exe" /NOLOGO "R:\abcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghij\abcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghij\abcdefghijabcdefghijabcdefghijabcdefghijababcdefghij\main.obj" /DEFAULTLIB:phobos64.lib /LIBPATH:"P:\app\dmd\lib64" /LIBPATH:"C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.22.27905\lib\x64" legacy_stdio_definitions.lib /LIBPATH:"C:\Program Files (x86)\Windows Kits\10\Lib\10.0.18362.0\ucrt\x64" /LIBPATH:"C:\Program Files (x86)\Windows Kits\10\lib\10.0.18362.0\um\x64"
LINK : fatal error LNK1181: 入力ファイル '.obj' を開けません。
R:\abcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghij\abcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghij\abcdefghijabcdefghijabcdefghijabcdefghijababcdefghij>"C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.22.27905\bin\HostX64\x64\link.exe" /NOLOGO "main.obj" /DEFAULTLIB:phobos64.lib /LIBPATH:"P:\app\dmd\lib64" /LIBPATH:"C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.22.27905\lib\x64" legacy_stdio_definitions.lib /LIBPATH:"C:\Program Files (x86)\Windows Kits\10\Lib\10.0.18362.0\ucrt\x64" /LIBPATH:"C:\Program Files (x86)\Windows Kits\10\lib\10.0.18362.0\um\x64"
LINK : fatal error LNK1181: 入力ファイル 'main.obj' を開けません。

I haven't tried newer versions or lld-link, but at least the 2019 link.exe doesn't appear to work.

@WebFreak001 WebFreak001 mentioned this issue Feb 13, 2023
Merged
@WebFreak001
Copy link
Member

fixed by #2589

shoo added a commit to shoo/dub that referenced this issue Mar 11, 2023
@shoo
Fix dlang#2443 git-dependency testcase is broken on Windows
03fe266
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@shoo @WebFreak001 @Geod24