Set install path of libSwiftScan to 'lib/swift/host' instead of platform-specific path string #62972

artemcm · 2023-01-11T17:12:02Z

No description provided.

artemcm · 2023-01-11T19:17:39Z

This is currently missing the piece that creates a symlink from the new location to the old location.

artemcm · 2023-01-12T00:40:22Z

compnerd

This likely will break the Windows packaging, and definitely will cause problems for Windows. A dynamically linked library that is not in CWD is a problem as it will not be found dynamically without altering the library search path.

compnerd · 2023-01-12T00:42:17Z

@swift-ci please build toolchain windows platform

compnerd · 2023-01-12T16:20:30Z

As expected, this breaks:

swift-ci@EC2AMAZ-JCVOGM7 C:\Users\swift-ci\jenkins\workspace\swift-PR-build-toolchain-windows>msbuild C:\Users\swift-ci\jenkins\workspace\swift-PR-build-toolchain-windows\swift-installer-scripts\platforms\Windows\toolchain.wixproj   -p:RunWixToolsOutOfProc=true   -p:OutputPath=T:\package\toolchain\   -p:IntermediateOutputPath=T:\package\toolchain\   -p:DEVTOOLS_ROOT=T:\Library\Developer\Toolchains\unknown-Asserts-development.xctoolchain   -p:TOOLCHAIN_ROOT=T:\Library\Developer\Toolchains\unknown-Asserts-development.xctoolchain 
Microsoft (R) Build Engine version 16.10.2+857e5a733 for .NET Framework
Copyright (C) Microsoft Corporation. All rights reserved.

Build started 1/12/2023 1:57:59 AM.
Project "C:\Users\swift-ci\jenkins\workspace\swift-PR-build-toolchain-windows\swift-installer-scripts\platforms\Windows\toolchain.wixproj" on node 1 (default targets).
PrepareForBuild:
  Creating directory "T:\package\toolchain\".
HarvestDirectory:
  C:\Program Files (x86)\WiX Toolset v3.11\bin\Heat.exe dir T:\Library\Developer\Toolchains\unknown-Asserts-development.xctoolchain\usr\lib\clang -cg ClangResources -dr _usr_lib_clang -scom -sreg -srd -var var.TOOLCHAIN_ROOT_USR_LIB_CLANG -nologo -gg -nologo -sfrag -out T:\package\toolchain\_ClangResources_dir.wxs
Compile:
  C:\Program Files (x86)\WiX Toolset v3.11\bin\candle.exe -dProductVersion=0.0.0 -dDEVTOOLS_ROOT=T:\Library\Developer\Toolchains\unknown-Asserts-development.xctoolchain -dTOOLCHAIN_ROOT=T:\Library\Developer\Toolchains\unknown-Asserts-development.xctoolchain -dTOOLCHAIN_ROOT_USR_LIB_CLANG=T:\Library\Developer\Toolchains\unknown-Asserts-development.xctoolchain\usr\lib\clang -dConfiguration=Debug -dOutDir=T:\package\toolchain\ -dPlatform=AnyCPU -dProjectDir=C:\Users\swift-ci\jenkins\workspace\swift-PR-build-toolchain-windows\swift-installer-scripts\platforms\Windows\ -dProjectExt=.wixproj -dProjectFileName=toolchain.wixproj -dProjectName=toolchain -dProjectPath=C:\Users\swift-ci\jenkins\workspace\swift-PR-build-toolchain-windows\swift-installer-scripts\platforms\Windows\toolchain.wixproj -dTargetDir=T:\package\toolchain\ -dTargetExt=.msi -dTargetFileName=toolchain.msi -dTargetName=toolchain -dTargetPath=T:\package\toolchain\toolchain.msi -out T:\package\toolchain\ -ext "C:\Program Files (x86)\WiX Toolset v3.11\bin\\WixUIExtension.dll" toolchain-amd64.wxs T:\package\toolchain\_ClangResources_dir.wxs
Link:
  C:\Program Files (x86)\WiX Toolset v3.11\bin\Light.exe -out T:\package\toolchain\toolchain.msi -pdbout T:\package\toolchain\toolchain.wixpdb -cultures:null -ext "C:\Program Files (x86)\WiX Toolset v3.11\bin\\WixUIExtension.dll" -contentsfile T:\package\toolchain\toolchain.wixproj.BindContentsFileListnull.txt -outputsfile T:\package\toolchain\toolchain.wixproj.BindOutputsFileListnull.txt -builtoutputsfile T:\package\toolchain\toolchain.wixproj.BindBuiltOutputsFileListnull.txt -wixprojectfile C:\Users\swift-ci\jenkins\workspace\swift-PR-build-toolchain-windows\swift-installer-scripts\platforms\Windows\toolchain.wixproj T:\package\toolchain\toolchain-amd64.wixobj T:\package\toolchain\_ClangResources_dir.wixobj
C:\Users\swift-ci\jenkins\workspace\swift-PR-build-toolchain-windows\swift-installer-scripts\platforms\Windows\toolchain-amd64.wxs(585): error LGHT0103: The system cannot find the file 'T:\Library\Developer\Toolchains\unknown-Asserts-development.xctoolchain\usr\lib\swift\windows\_InternalSwiftScan.lib'. [C:\Users\swift-ci\jenkins\workspace\swift-PR-build-toolchain-windows\swift-installer-scripts\platforms\Windows\toolchain.wixproj]
Done Building Project "C:\Users\swift-ci\jenkins\workspace\swift-PR-build-toolchain-windows\swift-installer-scripts\platforms\Windows\toolchain.wixproj" (default targets) -- FAILED.

Build FAILED.

"C:\Users\swift-ci\jenkins\workspace\swift-PR-build-toolchain-windows\swift-installer-scripts\platforms\Windows\toolchain.wixproj" (default target) (1) ->
(Link target) -> 
  C:\Users\swift-ci\jenkins\workspace\swift-PR-build-toolchain-windows\swift-installer-scripts\platforms\Windows\toolchain-amd64.wxs(585): error LGHT0103: The system cannot find the file 'T:\Library\Developer\Toolchains\unknown-Asserts-development.xctoolchain\usr\lib\swift\windows\_InternalSwiftScan.lib'. [C:\Users\swift-ci\jenkins\workspace\swift-PR-build-toolchain-windows\swift-installer-scripts\platforms\Windows\toolchain.wixproj]

    0 Warning(s)
    1 Error(s)

Time Elapsed 00:00:00.96

compnerd · 2023-01-12T16:20:45Z

Note that this will require some additional adjustments to swift-format as well.

artemcm · 2023-01-12T16:25:15Z

Note that this will require some additional adjustments to swift-format as well.

How is swift-format connected to this?

artemcm · 2023-01-12T16:28:54Z

As expected, this breaks:

@compnerd could you advise on what's going wrong here?
Looking at LLVMInstallSymlink it is supposed to do a copy operation on Windows:
https://github.com/nihaals/apple--llvm-project/blob/apple/main/llvm/cmake/modules/LLVMInstallSymlink.cmake

compnerd · 2023-01-12T16:58:36Z

Note that this will require some additional adjustments to swift-format as well.

How is swift-format connected to this?

IIRC that used _InternalSwiftScan? Or was it LSP?

@compnerd could you advise on what's going wrong here?

CMAKE_SHARED_LIBRARY_SUFFIX is not guaranteed to be correct, you should explicitly spell that out. Secondly, if this is now being built dynamically, you need to handle the shared runtime (_InternalSwiftScan.dll) AND the import library (_InternalSwiftScan.lib). The former is in bin the latter in lib. If this is being built statically (as it were), then it should be using the shared library suffix .lib and will be in lib. Additionally, it should then be prefixed with lib making the final name lib_InternalSwiftScan.lib. If this was converted to dynamic builds, then you will need to update the packaging manifest for the locations.

artemcm · 2023-01-12T17:02:33Z

Note that this will require some additional adjustments to swift-format as well.

How is swift-format connected to this?

IIRC that used _InternalSwiftScan? Or was it LSP?

Only swift-driver uses this library.

@compnerd could you advise on what's going wrong here?

CMAKE_SHARED_LIBRARY_SUFFIX is not guaranteed to be correct, you should explicitly spell that out. Secondly, if this is now being built dynamically, you need to handle the shared runtime (_InternalSwiftScan.dll) AND the import library (_InternalSwiftScan.lib). The former is in bin the latter in lib. If this is being built statically (as it were), then it should be using the shared library suffix .lib and will be in lib. Additionally, it should then be prefixed with lib making the final name lib_InternalSwiftScan.lib. If this was converted to dynamic builds, then you will need to update the packaging manifest for the locations.

This library has always only ever been built dynamically.
And the client used to get it from bin/_InternalSwiftScan.dll:
https://github.com/apple/swift-driver/blob/main/Sources/SwiftDriver/ExplicitModuleBuilds/ModuleDependencyScanning.swift#L502

compnerd · 2023-01-12T17:08:11Z

Only swift-driver uses this library.

Oh! Okay, I guess I was confusing the internal libraries :-(

This library has always only ever been built dynamically.
And the client used to get it from bin/_InternalSwiftScan.dll:
https://github.com/apple/swift-driver/blob/main/Sources/SwiftDriver/ExplicitModuleBuilds/ModuleDependencyScanning.swift#L502

Ah awesome! Then you need to symlink the dll and the import library.

artemcm · 2023-01-12T19:27:24Z

@swift-ci please build toolchain windows platform

artemcm · 2023-01-12T19:27:29Z

@swift-ci test

artemcm · 2023-01-12T21:07:16Z

The new errors don't seem related much:

C:\Users\swift-ci\jenkins\workspace\swift-PR-build-toolchain-windows\swift-corelibs-foundation\Tests\Foundation\Tests\TestURL.swift:562: error: TestURL.test_URLByResolvingSymlinksInPathShouldUseTheCurrentDirectory : XCTAssertEqual failed: ("file:///C:/Users/swift-ci/jenkins/workspace/swift-PR-build-toolchain-windows/build/tmp/org.swift.TestFoundation.TestURL.resourceValues.8820/foo/bar/baz") is not equal to ("file:///T:/tmp/org.swift.TestFoundation.TestURL.resourceValues.8820/foo/bar/baz") -

compnerd · 2023-01-13T00:24:55Z

@artemcm the Foundation failure isn't the issue - the artifacts not being found is the issue. That means that the installer failed to build.

compnerd · 2023-01-13T00:25:29Z

swift-ci@EC2AMAZ-JCVOGM7 C:\Users\swift-ci\jenkins\workspace\swift-PR-build-toolchain-windows>msbuild C:\Users\swift-ci\jenkins\workspace\swift-PR-build-toolchain-windows\swift-installer-scripts\platforms\Windows\toolchain.wixproj   -p:RunWixToolsOutOfProc=true   -p:OutputPath=T:\package\toolchain\   -p:IntermediateOutputPath=T:\package\toolchain\   -p:DEVTOOLS_ROOT=T:\Library\Developer\Toolchains\unknown-Asserts-development.xctoolchain   -p:TOOLCHAIN_ROOT=T:\Library\Developer\Toolchains\unknown-Asserts-development.xctoolchain 
Microsoft (R) Build Engine version 16.10.2+857e5a733 for .NET Framework
Copyright (C) Microsoft Corporation. All rights reserved.

Build started 1/12/2023 8:42:49 PM.
Project "C:\Users\swift-ci\jenkins\workspace\swift-PR-build-toolchain-windows\swift-installer-scripts\platforms\Windows\toolchain.wixproj" on node 1 (default targets).
PrepareForBuild:
  Creating directory "T:\package\toolchain\".
HarvestDirectory:
  C:\Program Files (x86)\WiX Toolset v3.11\bin\Heat.exe dir T:\Library\Developer\Toolchains\unknown-Asserts-development.xctoolchain\usr\lib\clang -cg ClangResources -dr _usr_lib_clang -scom -sreg -srd -var var.TOOLCHAIN_ROOT_USR_LIB_CLANG -nologo -gg -nologo -sfrag -out T:\package\toolchain\_ClangResources_dir.wxs
Compile:
  C:\Program Files (x86)\WiX Toolset v3.11\bin\candle.exe -dProductVersion=0.0.0 -dDEVTOOLS_ROOT=T:\Library\Developer\Toolchains\unknown-Asserts-development.xctoolchain -dTOOLCHAIN_ROOT=T:\Library\Developer\Toolchains\unknown-Asserts-development.xctoolchain -dTOOLCHAIN_ROOT_USR_LIB_CLANG=T:\Library\Developer\Toolchains\unknown-Asserts-development.xctoolchain\usr\lib\clang -dConfiguration=Debug -dOutDir=T:\package\toolchain\ -dPlatform=AnyCPU -dProjectDir=C:\Users\swift-ci\jenkins\workspace\swift-PR-build-toolchain-windows\swift-installer-scripts\platforms\Windows\ -dProjectExt=.wixproj -dProjectFileName=toolchain.wixproj -dProjectName=toolchain -dProjectPath=C:\Users\swift-ci\jenkins\workspace\swift-PR-build-toolchain-windows\swift-installer-scripts\platforms\Windows\toolchain.wixproj -dTargetDir=T:\package\toolchain\ -dTargetExt=.msi -dTargetFileName=toolchain.msi -dTargetName=toolchain -dTargetPath=T:\package\toolchain\toolchain.msi -out T:\package\toolchain\ -ext "C:\Program Files (x86)\WiX Toolset v3.11\bin\\WixUIExtension.dll" toolchain-amd64.wxs T:\package\toolchain\_ClangResources_dir.wxs
Link:
  C:\Program Files (x86)\WiX Toolset v3.11\bin\Light.exe -out T:\package\toolchain\toolchain.msi -pdbout T:\package\toolchain\toolchain.wixpdb -cultures:null -ext "C:\Program Files (x86)\WiX Toolset v3.11\bin\\WixUIExtension.dll" -contentsfile T:\package\toolchain\toolchain.wixproj.BindContentsFileListnull.txt -outputsfile T:\package\toolchain\toolchain.wixproj.BindOutputsFileListnull.txt -builtoutputsfile T:\package\toolchain\toolchain.wixproj.BindBuiltOutputsFileListnull.txt -wixprojectfile C:\Users\swift-ci\jenkins\workspace\swift-PR-build-toolchain-windows\swift-installer-scripts\platforms\Windows\toolchain.wixproj T:\package\toolchain\toolchain-amd64.wixobj T:\package\toolchain\_ClangResources_dir.wixobj
C:\Users\swift-ci\jenkins\workspace\swift-PR-build-toolchain-windows\swift-installer-scripts\platforms\Windows\toolchain-amd64.wxs(585): error LGHT0103: The system cannot find the file 'T:\Library\Developer\Toolchains\unknown-Asserts-development.xctoolchain\usr\lib\swift\windows\_InternalSwiftScan.lib'. [C:\Users\swift-ci\jenkins\workspace\swift-PR-build-toolchain-windows\swift-installer-scripts\platforms\Windows\toolchain.wixproj]
Done Building Project "C:\Users\swift-ci\jenkins\workspace\swift-PR-build-toolchain-windows\swift-installer-scripts\platforms\Windows\toolchain.wixproj" (default targets) -- FAILED.

Build FAILED.

"C:\Users\swift-ci\jenkins\workspace\swift-PR-build-toolchain-windows\swift-installer-scripts\platforms\Windows\toolchain.wixproj" (default target) (1) ->
(Link target) -> 
  C:\Users\swift-ci\jenkins\workspace\swift-PR-build-toolchain-windows\swift-installer-scripts\platforms\Windows\toolchain-amd64.wxs(585): error LGHT0103: The system cannot find the file 'T:\Library\Developer\Toolchains\unknown-Asserts-development.xctoolchain\usr\lib\swift\windows\_InternalSwiftScan.lib'. [C:\Users\swift-ci\jenkins\workspace\swift-PR-build-toolchain-windows\swift-installer-scripts\platforms\Windows\toolchain.wixproj]

    0 Warning(s)
    1 Error(s)

Time Elapsed 00:00:05.21

artemcm · 2023-01-13T18:57:20Z

@swift-ci please build toolchain windows platform

artemcm · 2023-01-13T19:43:04Z

@swift-ci please build toolchain windows platform

artemcm · 2023-01-13T23:03:19Z

@compnerd is there a away to get at the state of the build/install directory of this CI run?

…orm-specific path string

artemcm · 2023-01-13T23:26:10Z

@swift-ci please build toolchain windows platform

compnerd · 2023-01-13T23:30:38Z

@compnerd is there a away to get at the state of the build/install directory of this CI run?

I think that you can piece that together from the build logs, but I don't think that there is a way to do a directory listing (perhaps @shahmishal can help?)

artemcm · 2023-01-13T23:55:30Z

@swift-ci test

artemcm · 2023-01-14T00:44:10Z

@swift-ci please build toolchain windows platform

artemcm · 2023-01-14T02:50:03Z

@swift-ci test macOS platform

artemcm · 2023-01-17T17:11:52Z

@compnerd the current state of this change leaves the library in the same place on Windows. This should largely be fine since the client picks it up from bin/_InternalSwiftScan.dll anyways and is meant to be backwards-compatible with the current (before this change) path. Once I have a better Windows debug environment I can followup to unify the path handling.

artemcm mentioned this pull request Jan 11, 2023

[Dependency Scanning] Move computation of the path for libSwiftScan to the toolchain swiftlang/swift-driver#1253

Merged

artemcm force-pushed the MoveLibSwiftScanToHostDir branch 2 times, most recently from 7136f78 to 03bebef Compare January 12, 2023 00:39

artemcm marked this pull request as ready for review January 12, 2023 00:40

artemcm requested review from etcwilde, compnerd and DougGregor January 12, 2023 00:40

compnerd reviewed Jan 12, 2023

View reviewed changes

artemcm force-pushed the MoveLibSwiftScanToHostDir branch from 03bebef to e37aa78 Compare January 12, 2023 19:27

artemcm force-pushed the MoveLibSwiftScanToHostDir branch from e37aa78 to c16d3ae Compare January 13, 2023 18:57

Set install path of libSwiftScan to 'lib/swift/host' instead of platf…

d95cb95

…orm-specific path string

artemcm force-pushed the MoveLibSwiftScanToHostDir branch from c16d3ae to d95cb95 Compare January 13, 2023 23:25

artemcm merged commit 6c8c0b8 into swiftlang:main Jan 17, 2023

artemcm deleted the MoveLibSwiftScanToHostDir branch January 17, 2023 17:51

Set install path of libSwiftScan to 'lib/swift/host' instead of platform-specific path string #62972

Set install path of libSwiftScan to 'lib/swift/host' instead of platform-specific path string #62972

Uh oh!

Conversation

artemcm commented Jan 11, 2023

Uh oh!

artemcm commented Jan 11, 2023

Uh oh!

artemcm commented Jan 12, 2023

Uh oh!

compnerd left a comment

Choose a reason for hiding this comment

Uh oh!

compnerd commented Jan 12, 2023

Uh oh!

compnerd commented Jan 12, 2023

Uh oh!

compnerd commented Jan 12, 2023

Uh oh!

artemcm commented Jan 12, 2023

Uh oh!

artemcm commented Jan 12, 2023

Uh oh!

compnerd commented Jan 12, 2023

Uh oh!

artemcm commented Jan 12, 2023

Uh oh!

compnerd commented Jan 12, 2023

Uh oh!

artemcm commented Jan 12, 2023

Uh oh!

artemcm commented Jan 12, 2023

Uh oh!

artemcm commented Jan 12, 2023

Uh oh!

compnerd commented Jan 13, 2023

Uh oh!

compnerd commented Jan 13, 2023

Uh oh!

artemcm commented Jan 13, 2023

Uh oh!

artemcm commented Jan 13, 2023

Uh oh!

artemcm commented Jan 13, 2023

Uh oh!

artemcm commented Jan 13, 2023

Uh oh!

compnerd commented Jan 13, 2023

Uh oh!

artemcm commented Jan 13, 2023

Uh oh!

artemcm commented Jan 14, 2023

Uh oh!

artemcm commented Jan 14, 2023

Uh oh!

artemcm commented Jan 17, 2023

Uh oh!

Uh oh!