Swift test suite doesn't honour LIT_FILTER_OUT environment variable

[hjyamauchi]

Description

https://ci-external.swift.org/job/swift-main-windows-toolchain/1155/

********************
Failed Tests (1):
  Swift(android-aarch64) :: ClangImporter/availability_custom_domains.swift

********************
Unexpectedly Passed Tests (3):
  Swift(android-aarch64) :: Driver/parseable_output.swift
  Swift(android-aarch64) :: Driver/parseable_output_unicode.swift
  Swift-validation(android-aarch64) :: compiler_crashers_fixed/28795-inprotocol-isrequirementsignaturecomputed-missing-signature.swift

Reproduction

swift-main-windows-toolchain

Expected behavior

No failures

Environment

swift-main-windows-toolchain

Additional information

No response

[hjyamauchi]

Based on the change history #80040 might be cause.

[tshortli]

I marked that test unsupported on Windows, but apparently the Windows CI is testing Android:

# .---command stderr------------
# | <unknown>:0: warning: libc not found for 'aarch64-unknown-linux-android'; C stdlib may be unavailable
# | Assertion failed: Val && "isa<> used on a null pointer", file C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\llvm-project\llvm\include\llvm/Support/Casting.h, line 109
# | Please submit a bug report (https://swift.org/contributing/#reporting-bugs) and include the crash backtrace.
# | Stack dump:
# | 0.	Program arguments: t:\\5\\bin\\swift-frontend.exe -target aarch64-unknown-linux-android -sdk T:/android-ndk-r27c/toolchains/llvm/prebuilt/windows-x86_64/sysroot -resource-dir T:/aarch64-unknown-linux-android28/Runtime/./lib/swift -module-cache-path T:\\aarch64-unknown-linux-android28\\Runtime\\swift-test-results\\aarch64-unknown-linux-android\\clang-module-cache -swift-version 4 -define-availability "SwiftStdlib 9999:macOS 9999, iOS 9999, watchOS 9999, tvOS 9999" -define-availability "SwiftStdlib 5.0:macOS 10.14.4, iOS 12.2, watchOS 5.2, tvOS 12.2" -define-availability "SwiftStdlib 5.1:macOS 10.15, iOS 13.0, watchOS 6.0, tvOS 13.0" -define-availability "SwiftStdlib 5.2:macOS 10.15.4, iOS 13.4, watchOS 6.2, tvOS 13.4" -define-availability "SwiftStdlib 5.3:macOS 11.0, iOS 14.0, watchOS 7.0, tvOS 14.0" -define-availability "SwiftStdlib 5.4:macOS 11.3, iOS 14.5, watchOS 7.4, tvOS 14.5" -define-availability "SwiftStdlib 5.5:macOS 12.0, iOS 15.0, watchOS 8.0, tvOS 15.0" -define-availability "SwiftStdlib 5.6:macOS 12.3, iOS 15.4, watchOS 8.5, tvOS 15.4" -define-availability "SwiftStdlib 5.7:macOS 13.0, iOS 16.0, watchOS 9.0, tvOS 16.0" -define-availability "SwiftStdlib 5.8:macOS 13.3, iOS 16.4, watchOS 9.4, tvOS 16.4" -define-availability "SwiftStdlib 5.9:macOS 14.0, iOS 17.0, watchOS 10.0, tvOS 17.0" -define-availability "SwiftStdlib 5.10:macOS 14.4, iOS 17.4, watchOS 10.4, tvOS 17.4, visionOS 1.1" -define-availability "SwiftStdlib 6.0:macOS 15.0, iOS 18.0, watchOS 11.0, tvOS 18.0, visionOS 2.0" -define-availability "SwiftStdlib 6.1:macOS 9999, iOS 9999, watchOS 9999, tvOS 9999, visionOS 9999" -define-availability "SwiftStdlib 6.2:macOS 9999, iOS 9999, watchOS 9999, tvOS 9999, visionOS 9999" -typo-correction-limit 10 -enable-source-import -sdk C:\\\\Users\\\\swift-ci\\\\jenkins\\\\workspace\\\\swift-main-windows-toolchain\\\\swift\\\\test\\\\Inputs\\\\clang-importer-sdk -I C:\\\\Users\\\\swift-ci\\\\jenkins\\\\workspace\\\\swift-main-windows-toolchain\\\\swift\\\\test\\\\Inputs\\\\clang-importer-sdk\\\\swift-modules -typecheck -verify -import-objc-header C:\\Users\\swift-ci\\jenkins\\workspace\\swift-main-windows-toolchain\\swift\\test\\ClangImporter/Inputs/availability_domains_bridging_header.h -I C:\\Users\\swift-ci\\jenkins\\workspace\\swift-main-windows-toolchain\\swift\\test\\ClangImporter/Inputs/custom-modules/availability-domains -enable-experimental-feature CustomAvailability C:\\Users\\swift-ci\\jenkins\\workspace\\swift-main-windows-toolchain\\swift\\test\\ClangImporter\\availability_custom_domains.swift C:\\Users\\swift-ci\\jenkins\\workspace\\swift-main-windows-toolchain\\swift\\test\\ClangImporter/Inputs/availability_custom_domains_other.swift

[tshortli]

@hjyamauchi are you able to run the test and gather a backtrace I could use to understand the problem further? I don't have a way to reproduce Windows test failures and unfortunately the crash in the log from the bot is completely unsymbolicated so I have nothing to work with. I'll probably mark the test unsupported for Android, but it would be nice to see if I can reproduce the crash another way and fix it.

[tshortli]

Test disabled here #80060.

[hjyamauchi]

CC @weliveindetail

[hjyamauchi]

I think some Android tests are/were expected to fail, right? And the CI should pass now after disabling? I'd be happy to get a backtrace if it's still needed.

[tshortli]

If you can get a backtrace I'd appreciate it, but no rush.

[weliveindetail]

My PR this morning was supposed to fix that: #80053