"working directory" not completely respected when using "-explicit-swift-module-map-file"

[chiragramani]

Description
In our setup, -explicit-swift-module-map-file has relative paths. When the execution directory is the same as the working directory, it works all fine. When the execution directory is different, then it errors out.

error: emit-module command failed with exit code 1 (use -v to see invocation)
<unknown>:0: error: failed to open explicit Swift module: outputs/StressTest1.swiftmodule
.../swift_repro/foo.swift:5:8: error: no such module 'StressTest1'
import StressTest1

Though we can have absolute paths in "-explicit-swift-module-map-file" which would fix the above error, but we would like to have relative paths given remote caching and reproducibility aspects involved.

Steps to reproduce

1. Download swift_repro.zip.
2. ./root.sh - note - the first part of the script will be successful (execution directory the same as working directory)
3. The second part will fail.
4. Now - execute ./make_paths_absolute.sh - this makes the path in the "-explicit-swift-module-map-file" absolute.
5. Now, execute ./root.sh again and this builds fine now.

Expected behavior

1. working directory should be honoured(if present) when resolving paths mentioned in the -explicit-swift-module-map-file.

Environment

• Swift compiler version info - swift-driver version: 1.75.2 Apple Swift version 5.8.1 (swiftlang-5.8.0.124.5 clang-1403.0.22.11.100)
• Xcode version info - Xcode 14.3.1, Build version 14E300b
• Deployment target: iOS 15

Note: my understanding of the expected behavior might be wrong or there might be something else going on, so please feel free to correct me or share other ways to go about it, whatever you think is ideal.

[chiragramani]

cc: @artemcm @cachemeifyoucan

[cachemeifyoucan]

Thanks for reporting. We can probably make explicit module map to work with relative path but I am also interested in why the project is setup using -explicit-swift-module-map-file. That file is usually generated by swift driver using dependency scanner when explicit module build is enabled.

A much human friendly to specify that in your implicit module build is just replace the map file with -I $WORKING_DIRECTORY/outputs.

[chiragramani]

but I am also interested in why the project is setup using -explicit-swift-module-map-file

Currently, rules_swift constructs this file when the explicit module build feature is enabled. cc: @keith @brentleyjones as they can best share the related context involved.

[brentleyjones]

We use the map file because the outputs in a bazel build are all to different directories. Using multiple -I statements has proven in the past to be a performance issue.

The relative paths seem to work, only if they are relative to the executing directory. With bazel this is fine, but we have an IDE integration that relies on -working-directory (which is different from pwd), which currently breaks per this bug report.

So if you could make the relative paths work based on -working-directory, it would really help us out 🙏.

[brentleyjones]

That file is usually generated by swift driver using dependency scanner when explicit module build is enabled.

Also of note, bazel builds will most likely want to alway side-step the dependency scanner, and handle dependencies itself.

[cachemeifyoucan]

I understand your use case and I understand the reason for such decision. I do wonder how slow using -I is, giving the directory it points to is not enormous.

The fundamental issue here is that swift-frontend has no notion of working directory. -working-directory is a flag that is only consumed by swift-driver, and swift-frontend just ignores that. SwiftDriver is in charged of canonicalizing all the paths for swift-frontend and maybe passing working directory via -Xcc flag to embedded clang because clang does care about working directory.

I don't think I want to make a call if we want to change that model or not. Maybe @artemcm is the better person to make a decision here.

The other issue for current bezel workflow is that it is not handling the module dependencies (?) from swift/system. Can you clarify if that is the case? The future direction is to get rid of this world of mixed implicit/explicit module.

[chiragramani]

I do wonder how slow using -I is, giving the directory it points to is not enormous.

The repro enclosed in this issue is a basic setup demonstrating the problem with relative paths not working and given the smaller scope of build involved, we don't see slowness with -I in this specific repro but we see implicit module builds being extremely slow for our core applications (Rider, Eats and Driver). The core targets depend on 400/500 mixed bag of Swift, ObjC libraries, and Frameworks. When we tap on a symbol, it takes 40-50+ seconds for Xcode to respond and navigate to the right file (having the symbol definition). On profiling this, we find 70%-80% of the time is spent in doing stat calls where we believe, explicit module builds will help, hence we are trying to get this work. (though vfs overlay and a bunch of other optimizations can also help)

SourceKit logs
Notice the request and response timestamp for a "view symbol definition action"

2023-06-20 13:15:06.005 Xcode[8547:10485338] SourceKit-client: [2:request:107387:945.0651] [341] {

2023-06-20 13:16:17.027 Xcode[8547:10485782] SourceKit-client: [2:response:156007:945.0895] [341] {

Maybe @artemcm is the better person to make a decision here.

@artemcm what are your thoughts here?

The other issue for current bezel workflow is that it is not handling the module dependencies (?) from swift/system. Can you clarify if that is the case? The future direction is to get rid of this world of mixed implicit/explicit module.

The Bazel rules explicitly call out all the required dependencies from Swift/system (pointing to appropriate search paths, SDKs, and other deps involved). https://github.com/bazelbuild/rules_swift also has a well-curated set of few examples where you can do a bazel aquery ... to know all the params/actions that go in order to compile a module or build a bundle if we would like to confirm the exec context involved. If you would like me to do it for a specific case, please let me know and I will get those details as well.

[cachemeifyoucan]

Thanks for the data. That is helpful. I do wonder how many syscalls we can reduce comparing:

• Using explicit module
• Loading binary module from -I option

The Bazel rules explicitly call out all the required dependencies from Swift/system (pointing to appropriate search paths, SDKs, and other deps involved).

I am not familiar with Bazel and never played with it. I asked the question because from the example attached, the explicit swift module map only contains one module, which I am not sure if that is the case for production system. Ideally, we want all the dependencies are exhaustively listed in there and we can use -disable-implicit-swift-modules. I will check bazel aquery when I have time.

As to support working-directory in swift-frontend, I don't have the context for all the reason behind such decision. I can guess a few, for example, the underlying file system is shared by swift and clang, also between all the implicit build threads. In the explicit build work, the worst problem is gone, but it is hard to argue against the simplicity of resolving all the path in driver so swift-frontend doesn't need to deal with that. I can't make a call for that and I am also willing to learn all the reasons behind that for a discussion.

[keith]

Bazel is using the feature added here to allow us to use the explicit module map feature, while still enabling implicit modules for system dependencies, during this transition time before we support fully explicit module builds. So in bazel it's expected that today that file only contains modules from your project, as the system modules are still discovered implicitly.

As Brentley said above the original intent of this was that using -I many times for many search paths had a quadratic performance impact and this approach improved compilation time for us by ~30%! (more context here) I believe the compilation performance issue is actually fixed today but it sounds like Xcode might still suffer from a similar performance bottleneck (maybe if this was working we could test that more accurately).

Aside from performance there is another hermeticity benefit from using this file as opposed to -I. The case is if you have a directory with multiple .swiftmodule files like this:

DerivedData/.../Foo.swiftmodule
DerivedData/.../Bar.swiftmodule

And you want to be able to depend on Foo but not Bar, using -I does not allow you to restrict this, but if you use explicit modules and exclude Bar from the explicit module json file you can limit this. Hermeticity is generally a core feature of bazel so this is a benefit for us where we require users to include Foo and Bar in their deps if they want to depend on both (there are some caveats to this with using sandboxing but some folks disable that because of the performance impact).

[DavidGoldman]

The Swift compiler does some weird stuff with the working directory argument like manually applying it to all path args, see here. This looks like an oversight - the working dir is unused https://github.com/apple/swift/blob/77e694ad0d3729f4a5cab48d3cfcb3e0d34e5f74/lib/Frontend/CompilerInvocation.cpp#L1748C10-L1748C32 and the parser has no idea: https://github.com/apple/swift/blob/77e694ad0d3729f4a5cab48d3cfcb3e0d34e5f74/include/swift/Frontend/ModuleInterfaceLoader.h#L318

[cachemeifyoucan]

The working directory code handling code you point out is in Driver (which is going to be removed and is already replaced by swift-driver). The idea is for the swift driver to canonicalize all the paths so swift-frontend do not need to worry about working directory.

[DavidGoldman]

AFAIK sourcekitd is still using the C++ driver, right, not swift-driver?