Avoid locale dependent <ctype.h> functions like isascii(), isdigit(), tolower()

[vstinner]

Feature or enhancement

Has this already been discussed elsewhere?

No response given

Links to previous discussion of this feature:

No response

Proposal:

The following C files use <ctype.h> functions which depend on the current LC_CTYPE locale:

$ git grep -l -E '\b(isalnum|isalpha|iscntrl|isdigit|islower|isgraph|isprint|ispunct|isspace|isupper|isxdigit|tolower|toupper)\b'|grep -E '\.c$'
Modules/_decimal/libmpdec/io.c
Modules/_sre/sre.c
Modules/_zoneinfo.c
Modules/getaddrinfo.c
Objects/bytearrayobject.c
Objects/bytes_methods.c
Objects/bytesobject.c
Objects/unicodeobject.c
PC/launcher.c
PC/launcher2.c
Parser/tokenizer.c
Python/formatter_unicode.c
Python/pystrcmp.c

I propose to replace them with Python C API functions which don't depend on the locale, like Py_ISDIGIT() and Py_TOLOWER().

Linked PRs

• gh-108767: Convert Py_ISLOWER() macro to static inline function #108770
• gh-108767: Replace ctype.h functions with pyctype.h functions #108772

[serhiy-storchaka]

The use in Modules/_sre/sre.c is intentional.

[vstinner]

Modules/_decimal/libmpdec/io.c is a copy of libmpdec, I prefer to leave it unchanged.

I also prefer to leave the Windows launcher program unchanged:

PC/launcher.c
PC/launcher2.c

[vstinner]

By the way, pyport.h has an interesting code:

/* On 4.4BSD-descendants, ctype functions serves the whole range of
 * wchar_t character set rather than single byte code points only.
 * This characteristic can break some operations of string object
 * including str.upper() and str.split() on UTF-8 locales.  This
 * workaround was provided by Tim Robbins of FreeBSD project.
 */

#if defined(__APPLE__)
#  define _PY_PORT_CTYPE_UTF8_ISSUE
#endif

#ifdef _PY_PORT_CTYPE_UTF8_ISSUE
#ifndef __cplusplus
   /* The workaround below is unsafe in C++ because
    * the <locale> defines these symbols as real functions,
    * with a slightly different signature.
    * See issue #10910
    */
#include <ctype.h>
#include <wctype.h>
#undef isalnum
#define isalnum(c) iswalnum(btowc(c))
#undef isalpha
#define isalpha(c) iswalpha(btowc(c))
#undef islower
#define islower(c) iswlower(btowc(c))
#undef isspace
#define isspace(c) iswspace(btowc(c))
#undef isupper
#define isupper(c) iswupper(btowc(c))
#undef tolower
#define tolower(c) towlower(btowc(c))
#undef toupper
#define toupper(c) towupper(btowc(c))
#endif
#endif