File Hashing for Incremental Builds

[benb]

This PR:

• Adds a command line option --enable-incremental-file-hashing, and a corresponding --disable-incremental-file-hashing, defaulting to disabled
  • if this PR is accepted, this will need a corresponding pull request to the main swift repo that introduces the option to Options.inc
• When file-hashing is enabled, both the source files and external dependency files are hashed with SHA256
• Those hashes are used to verify that a file is unchanged even if the timestamp is updated
• They are serialized with the rest of the ModuleDependencyGraph info in the blob field
• The serialized version is given a minor version bump to account for this change
• Tests added

RFC discussion here: https://forums.swift.org/t/rfc-file-hashes-for-swift-incremental-compilation/

[drodriguez]

@swift-ci please smoke test

Broken tree state because swiftlang/swift#82153 had not landed yet.

[drodriguez]

@swift-ci please smoke test

[owenv]

LGTM, just had a few minor comments

[owenv]

Could this be marked @_spi(Testing)? I don't think it needs to be API

[benb]

I'm afraid this needs to be exposed in the DriverExecutor protocol, so I think it needs to be public.

[owenv]

Ah, I forgot this showed up in that API. We may need a strategy to stage in the changes to the DriverExecutor protocol so they don't break https://github.com/swiftlang/swift-build, or make synchronized changes to the repos. Maybe you could keep the old execute requirements around temporarily for default implementations of the new ones to call?

[benb]

Ah, I didn't realise there was an API here exposed to swift-build.

I've added a commit that restores the original methods to DriverExecutor, with an extension that gives default implementations of the new methods, and added in definitions with fatalError() if these legacy methods are called on the implementations defined in swift-driver itself.

3ecc734

[benb]

I had to fix the added tests - the old logic stopped working after rebasing this branch past 7c709c5

Honestly, I'm not sure my tests are the right approach, however the majority of tests in IncrementalCompilationTests are failing for me locally on main, so I don't want to copy those approaches directly. Maybe I am doing something wrong locally? Most other tests are passing for me, though.

[artemcm]

however the majority of tests in IncrementalCompilationTests are failing for me locally on main, so I don't want to copy those approaches directly.

What kinds of failures are you seeing? I recommend building and running tests against most-recent main snapshot from https://www.swift.org/install/macos/ or a locally-built ToT main of the compiler.

[benb]

however the majority of tests in IncrementalCompilationTests are failing for me locally on main, so I don't want to copy those approaches directly.

What kinds of failures are you seeing? I recommend building and running tests against most-recent main snapshot from https://www.swift.org/install/macos/ or a locally-built ToT main of the compiler.

Test Suite 'IncrementalCompilationTests' failed at 2025-06-13 17:30:25.064. Executed 37 tests, with 31 failures (0 unexpected) in 49.195 (49.197) seconds

I'll try with a snapshot.

[benb]

I'll try with a snapshot.
Using swiftly main-snapshot on a commit from yesterday:

$ ~/.swiftly/bin/swift --version
Apple Swift version 6.2-dev (LLVM d266fd9f80a7945, Swift 665515c781999a8)
Target: arm64-apple-macosx15.0
Build config: +assertions

I'm still getting lots of failures in IncrementalCompilationTests on current main (8c130e3):

$ ~/.swiftly/bin/swift test
…
Test Suite 'IncrementalCompilationTests' failed at 2025-06-13 19:33:51.708.
         Executed 34 tests, with 31 failures (0 unexpected) in 132.160 (132.161) seconds
…

Although I see strictly speaking there are '0 unexpected' failures, but I'm not sure how or why these are defined as being 'expected'.