`-Werror=unguarded-availability-new` in sysconfig prevents building of `watchdog`

[asottile-sentry]

admittedly this is pointing out a real warning in watchdog -- but it makes it not installable using specifically these pythons:

wget https://github.com/indygreg/python-build-standalone/releases/download/20240107/cpython-3.12.1+20240107-x86_64-apple-darwin-pgo+lto-full.tar.zst
tar -xf cpython-3.12.1+20240107-x86_64-apple-darwin-pgo+lto-full.tar.zst
./python/install/bin/python3 -m pip wheel watchdog --no-deps

output:

...
      running build_ext
      building '_watchdog_fsevents' extension
      creating build/temp.macosx-10.9-x86_64-cpython-312
      creating build/temp.macosx-10.9-x86_64-cpython-312/src
      clang -fno-strict-overflow -DNDEBUG -g -O3 -Wall -arch x86_64 -mmacosx-version-min=10.9 -Wno-nullability-completeness -Wno-expansion-to-defined -Wno-undef-prefix -fPIC -Werror=unguarded-availability-new -DWATCHDOG_VERSION_STRING=\"3.0.0\" -DWATCHDOG_VERSION_MAJOR=3 -DWATCHDOG_VERSION_MINOR=0 -DWATCHDOG_VERSION_BUILD=0 -I/private/tmp/y/qq/python/install/include/python3.12 -c src/watchdog_fsevents.c -o build/temp.macosx-10.9-x86_64-cpython-312/src/watchdog_fsevents.o -std=c99 -pedantic -Wall -Wextra -fPIC -Wno-nullability-completeness -Wno-nullability-extension -Wno-newline-eof -Wno-error=unused-command-line-argument
      src/watchdog_fsevents.c:191:25: error: 'kFSEventStreamEventFlagItemCloned' is only available on macOS 10.13 or newer [-Werror,-Wunguarded-availability-new]
      FLAG_PROPERTY(IsCloned, kFSEventStreamEventFlagItemCloned)
                              ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      src/watchdog_fsevents.c:163:27: note: expanded from macro 'FLAG_PROPERTY'
              if (self->flags & flag) { \
                                ^~~~
      /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/System/Library/Frameworks/CoreServices.framework/Frameworks/FSEvents.framework/Headers/FSEvents.h:544:3: note: 'kFSEventStreamEventFlagItemCloned' has been marked as being introduced in macOS 10.13 here, but the deployment target is macOS 10.9.0
        kFSEventStreamEventFlagItemCloned __OSX_AVAILABLE_STARTING(__MAC_10_13, __IPHONE_11_0) = 0x00400000
        ^
      src/watchdog_fsevents.c:191:25: note: enclose 'kFSEventStreamEventFlagItemCloned' in a __builtin_available check to silence this warning
      FLAG_PROPERTY(IsCloned, kFSEventStreamEventFlagItemCloned)
                              ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      src/watchdog_fsevents.c:163:27: note: expanded from macro 'FLAG_PROPERTY'
              if (self->flags & flag) { \
                                ^~~~
      src/watchdog_fsevents.c:596:40: error: 'kFSEventStreamCreateFlagUseExtendedData' is only available on macOS 10.13 or newer [-Werror,-Wunguarded-availability-new]
                                           | kFSEventStreamCreateFlagUseExtendedData
                                             ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/System/Library/Frameworks/CoreServices.framework/Frameworks/FSEvents.framework/Headers/FSEvents.h:294:3: note: 'kFSEventStreamCreateFlagUseExtendedData' has been marked as being introduced in macOS 10.13 here, but the deployment target is macOS 10.9.0
        kFSEventStreamCreateFlagUseExtendedData __OSX_AVAILABLE_STARTING(__MAC_10_13, __IPHONE_11_0) = 0x00000040,
        ^
      src/watchdog_fsevents.c:596:40: note: enclose 'kFSEventStreamCreateFlagUseExtendedData' in a __builtin_available check to silence this warning
                                           | kFSEventStreamCreateFlagUseExtendedData
                                             ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      2 errors generated.
      error: command '/usr/bin/clang' failed with exit code 1
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for watchdog
Failed to build watchdog
ERROR: Failed to build one or more wheels

[notice] A new release of pip is available: 23.3.2 -> 24.0
[notice] To update, run: /private/tmp/y/qq/python/install/bin/python3 -m pip install --upgrade pip

a workaround is to remove that flag from _sysconfigdata__darwin_darwin.py after unpacking

[indygreg]

Assuming you are trying to build watchdog binaries that work on macOS <10.13, this compiler error is absolutely finding a bug in watchdog where it is failing to conditionally guard symbols not available in older macOS versions!

On one hand I consider this a feature: we're preventing bugs!

On the other, we are inserting -Werror=unguarded-availability-new into the compiler flags used to build CPython itself and CPython's build system exposes that as a CFLAG that should be used to compile extension modules. This propagation is definitely an unintended side-effect. (We added this compiler flag as a guardrail to help ensure our CPython builds don't reference symbols that aren't available in the target macOS version. This is a best practice when building portable macOS binaries. TBH I'm kind of surprised the CPython builds don't do this. I can make a case that extension module builds should be using this flag to prevent end-users from building wheels that don't conform to stated ABI compatibility.)

I think it is reasonable to strip this - and other custom - compiler flag from the sysconfig data to maintain compatibility with CPython.

[charliermarsh]

Is this roughly: add some additional replacements to hack_sysconfig.py (in build_cpython.sh) to strip out -Werror=unguarded-availability-new?

(Do we also need to strip from CPPFLAGS?)

[indygreg]

Yeah, this is a modification to hack_sysconfig.py to strip flags/words from variables.

It looks like the flag is present in the following:

• CFLAGS
• CONFIGURE_CFLAGS
• CONFIGURE_CPPFLAGS
• CONFIG_ARGS
• CPPFLAGS
• PY_BUILTIN_MODULE_CFLAGS
• PY_CFLAGS
• PY_CORE_CFLAGS
• PY_CPPFLAGS
• PY_STDMODULE_CFLAGS

I'd have to look at which variables are used by Python build tools to propagate compiler flags: those are the ones we want to modify.

I'm hesitant to modify flags that denote how the Python distribution itself was built: that would constitute lying. But if a variable is overloaded to both reflect what was used and what to use, then I suppose we must modify it.

[frou]

I'm encountering this error when trying to use uv-managed Python to develop Google Cloud Functions (functions-framework depends on watchdog)

[charliermarsh]

At least in setuptools I see:

if 'CC' in os.environ:
    newcc = os.environ['CC']
    if 'LDSHARED' not in os.environ and ldshared.startswith(cc):
        # If CC is overridden, use that as the default
        #       command for LDSHARED as well
        ldshared = newcc + ldshared[len(cc) :]
    cc = newcc
if 'CXX' in os.environ:
    cxx = os.environ['CXX']
if 'LDSHARED' in os.environ:
    ldshared = os.environ['LDSHARED']
if 'CPP' in os.environ:
    cpp = os.environ['CPP']
else:
    cpp = cc + " -E"  # not always
if 'LDFLAGS' in os.environ:
    ldshared = ldshared + ' ' + os.environ['LDFLAGS']
if 'CFLAGS' in os.environ:
    cflags = cflags + ' ' + os.environ['CFLAGS']
    ldshared = ldshared + ' ' + os.environ['CFLAGS']
if 'CPPFLAGS' in os.environ:
    cpp = cpp + ' ' + os.environ['CPPFLAGS']
    cflags = cflags + ' ' + os.environ['CPPFLAGS']
    ldshared = ldshared + ' ' + os.environ['CPPFLAGS']
if 'AR' in os.environ:
    ar = os.environ['AR']
if 'ARFLAGS' in os.environ:
    archiver = ar + ' ' + os.environ['ARFLAGS']
else:
    archiver = ar + ' ' + ar_flags

So, that would be CFLAGS and CPPFLAGS?