5.38 outputs locale errors with UTF-8 LANG environment variable set

[LordAro]

Any execution of perl 5.38 with a UTF-8 language set results in

~/Downloads/strawberry-perl-5.38.0.1-64bit-portable/perl/bin/perl -v | head -2
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
        LC_ALL = (unset),
        (possibly more locale environment variables)
        LANG = "en_GB.UTF-8"
    are supported and installed on your system.
perl: warning: Falling back to the system default locale ("English_United Kingdom.1252").

This is perl 5, version 38, subversion 0 (v5.38.0) built for MSWin32-x64-multi-thread

it's not quite clear what languages are supported and installed on my system, and why this is only an issue now.

There's a few locale related things in 5.38, which it appears to be related to:

https://perldoc.perl.org/perl5380delta#Locale-category-LC_NAME-now-supported-on-participating-platforms
https://perldoc.perl.org/perl5380delta#Changes-to-%22thread-safe%22-locales

though I've no idea which.

Codepage is 850, but there's no difference if I set it to 65001 either.

Happens in both powershell and msys, and with either of LANG & LC_NAME - which would definitely suggest that it's perl at fault rather than MSYS / powershell.

There is a workaround, but it's hacky:

$ PERL_BADLANG=0 ~/Downloads/strawberry-perl-5.38.0.1-64bit-portable/perl/bin/perl -v | head -2

This is perl 5, version 38, subversion 0 (v5.38.0) built for MSWin32-x64-multi-thread

No issues with 5.32 or 5.36.

This feels related to, but distinct from, #119 . I have tried 5.38.0.2, it had no effect.

[shawnlaffan]

These are the same symptoms as in #97

The workaround for that (not documented in that issue) was to ensure LC_ALL was set.

See also Perl/perl5#21567

[shawnlaffan]

Also, are you running under docker?

[LordAro]

setting LC_ALL doesn't seem to work for me?

$ LC_ALL=en_GB.UTF-8 LANG= ~/Downloads/strawberry-perl-5.38.0.1-64bit-portable/perl/bin/perl -v | head -2
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
        LC_ALL = "en_GB.UTF-8",
        (possibly more locale environment variables)
        LANG = (unset)
    are supported and installed on your system.
perl: warning: Falling back to the system default locale ("English_United Kingdom.1252").

This is perl 5, version 38, subversion 0 (v5.38.0) built for MSWin32-x64-multi-thread

$ LC_ALL=en_GB.UTF-8 LANG=en_GB.UTF-8 ~/Downloads/strawberry-perl-5.38.0.1-64bit-portable/perl/bin/perl -v | head -2
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
        LC_ALL = "en_GB.UTF-8",
        (possibly more locale environment variables)
        LANG = "en_GB.UTF-8"
    are supported and installed on your system.
perl: warning: Falling back to the system default locale ("English_United Kingdom.1252").

This is perl 5, version 38, subversion 0 (v5.38.0) built for MSWin32-x64-multi-thread

I'm not running in docker (yet..), just a standard MSYS2 UCRT64 terminal

[shawnlaffan]

What happens if you run using a Cmd shell?

Run the portableshell.bat file at the top of the Strawberry Perl distribution to set things up.

[LordAro]

No changes with or without portableshell.bat

Without:

C:\Windows\System32>C:\Users\cpigott\Downloads\strawberry-perl-5.38.0.1-64bit-portable\perl\bin\perl.exe -v

This is perl 5, version 38, subversion 0 (v5.38.0) built for MSWin32-x64-multi-thread
...

C:\Windows\System32>set LANG=en_GB.UTF-8

C:\Windows\System32>C:\Users\cpigott\Downloads\strawberry-perl-5.38.0.1-64bit-portable\perl\bin\perl.exe -v
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
        LC_ALL = (unset),
        (possibly more locale environment variables)
        LANG = "en_GB.UTF-8"
    are supported and installed on your system.
perl: warning: Falling back to the system default locale ("English_United Kingdom.1252").

This is perl 5, version 38, subversion 0 (v5.38.0) built for MSWin32-x64-multi-thread
...

C:\Windows\System32>set LC_ALL=en_GB.UTF-8

C:\Windows\System32>C:\Users\cpigott\Downloads\strawberry-perl-5.38.0.1-64bit-portable\perl\bin\perl.exe -v
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
        LC_ALL = "en_GB.UTF-8",
        (possibly more locale environment variables)
        LANG = "en_GB.UTF-8"
    are supported and installed on your system.
perl: warning: Falling back to the system default locale ("English_United Kingdom.1252").

This is perl 5, version 38, subversion 0 (v5.38.0) built for MSWin32-x64-multi-thread
...

With:

C:\Windows\System32>C:\Users\cpigott\Downloads\strawberry-perl-5.38.0.1-64bit-portable\portableshell.bat
----------------------------------------------
 Welcome to Strawberry Perl Portable Edition!
 * URL - http://www.strawberryperl.com/
 * see README.TXT for more info
----------------------------------------------
Perl executable: C:\Users\cpigott\Downloads\strawberry-perl-5.38.0.1-64bit-portable\perl\bin\perl.exe
Perl version   : 5.38.0 / MSWin32-x64-multi-thread

C:\Windows\System32>perl -v

This is perl 5, version 38, subversion 0 (v5.38.0) built for MSWin32-x64-multi-thread
...

C:\Windows\System32>set LANG=en_GB.UTF-8

C:\Windows\System32>perl -v
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
        LC_ALL = (unset),
        (possibly more locale environment variables)
        LANG = "en_GB.UTF-8"
    are supported and installed on your system.
perl: warning: Falling back to the system default locale ("English_United Kingdom.1252").

This is perl 5, version 38, subversion 0 (v5.38.0) built for MSWin32-x64-multi-thread
...

C:\Windows\System32>set LC_ALL=en_GB.UTF-8

C:\Windows\System32>perl -v
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
        LC_ALL = "en_GB.UTF-8",
        (possibly more locale environment variables)
        LANG = "en_GB.UTF-8"
    are supported and installed on your system.
perl: warning: Falling back to the system default locale ("English_United Kingdom.1252").

This is perl 5, version 38, subversion 0 (v5.38.0) built for MSWin32-x64-multi-thread
...

[shawnlaffan]

Thanks for testing. I suspect this is an issue in the perl core and might have been addressed in the 5.39 development series.

Just as a final check, what happens if you change the code page in the Cmd prompt before calling perl?

For example for "plain" UTF-8:
chcp 65001

[LordAro]

No change

[sisyphus]

I think this is just part of the general poor support for UTF-8 locales with mingw-w64 msvcrt runtimes.
On Strawberry-5.38.0:

C:\>perl -MPOSIX -wle "$loc = POSIX::setlocale( LC_ALL, 'en_GB.UTF-8' ); print $loc;"
Use of uninitialized value $loc in print at -e line 1.

Same system, perl-5.38.0 built by Visual Studio 2022:

C:\>perl -MPOSIX -wle "$loc = POSIX::setlocale( LC_ALL, 'en_GB.UTF-8' ); print $loc;"
en_GB.UTF-8

Same system, perl-5.39.3, built mingw-w64 port of gcc-13.2.0 ( "ucrt" build, not "msvcrt" build):

C:\>perl -MPOSIX -wle "$loc = POSIX::setlocale( LC_ALL, 'en_GB.UTF-8' ); print $loc;"
en_GB.UTF-8

It's tempting to therefore think that this issue has been fixed in perl-5.39.3.
But I think it works with 5.39.3 only because that perl was built with a mingw-w64 UCRT compiler.

I haven't tried building perl-5.38.0 with that same UCRT mingw-w64 compiler.
I think it requires a patch or two.
IIRC, 5.39.2 was the first release that could be built straight out of the box with a mingw-w64 UCRT compiler.
If you build perl-5.39.3 using a mingw-w64 MSVCRT compiler, I think you'll still hit the same problem with the en_GB.UTF-8 locale. (But I'm not 100% sure about that.)

Cheers,
Rob

[shawnlaffan]

I think another batch of locale changes landed in time for perl 5.39.4, which was released yesterday. Would be worth testing.

[LordAro]

I'm game. ...how do I do that? :)

[shawnlaffan]

For a basic vanilla perl, which is all that is needed here, you can adapt the instructions in #39 (comment)

Alternately you could run with the strawberry perl infrastructure but delete any of the CPAN entries to save time from the config file. The one for 5.38 is at https://github.com/StrawberryPerl/Perl-Dist-Strawberry/blob/master/share/64bit-5.38.0.1.pp .

Clone this repo, then build the Perl::Dist::Strawberry module using whichever perl you like (edit - whichever strawberry perl). Then adapt the script at https://github.com/StrawberryPerl/Perl-Dist-Strawberry/blob/master/devel.utils/_build-5.38.bat to refer to the correct locations.

[sisyphus]

You'll need a UCRT compiler - eg from https://winlibs.com.
Strawberry Perl compilers are MSVCRT, and therefore (apparently) suffer the deficiency.

On a perl-5.39.4 built using gcc version 13.2.0 (MinGW-W64 x86_64-ucrt-mcf-seh, built by Brecht Sanders) I get:

D:\>perl -MPOSIX -wle "print $]; $loc = POSIX::setlocale( LC_ALL, 'en_GB.UTF-8' ); print $loc;"
5.039004
en_GB.UTF-8

But the output of that one-liner reverts to:

5.039004
Use of uninitialized value $loc in print at -e line 1.

when perl-5.39.4 is built using gcc version 13.2.0 (MinGW-W64 x86_64-msvcrt-posix-seh, built by Brecht Sanders).

Cheers,
Rob

[shawnlaffan]

Well that answers my suggestion that recent perl changes might fix the issue for MSVCRT compilers.

So the next question is whether the MSVCRT limitations can be worked around?

[sisyphus]

So the next question is whether the MSVCRT limitations can be worked around?

I don't know.
I haven't really thought beyond "there's something missing from this runtime".

The latest gcc-13.2.0 compilers have boosted the runtime version from, 11.0.0 to 11.0.1. Perhaps (unlikely ?) they address this issue. (I haven't tested.)

There was a request for submissions re the backporting of stuff into 5.38.1.
You could request that the option of building perl-5.38.1 with a UCRT compiler be provided.
But there's probably not much point in that if StrawberryPerl is committed to staying with the current gcc toolchain for the 5.38.x cycle.
I actually think you would find that if 5.38.1 was buildable using a UCRT compiler, then you could switch to a 13.1.0-UCRT compiler without any need to rebuild any of the 3rd party libraries that are provided, (But this is, of course, unproven and might even be seen as obviously silly to a more knowledgeable person.)

For 5.40 onwards, I think that using a UCRT compiler will be the thing to do.

BTW, I have just checked that 5.38.0 doesn't build with a UCRT compiler - and it doesn't.

....
gcc -c -I.\include -I. -I.. -DWIN32 -DWIN64 -DPERLDLL -DPERL_CORE -Os -D__USE_MINGW_ANSI_STDIO -fwrapv -fno-strict-aliasing -DPERL_IS_MINIPERL -omini\fcrypt.o fcrypt.c
In file included from ..\perl.h:3783,
from ..\perlio.c:43:
..\perlio.c: In function 'PerlIOStdio_invalidate_fileno':
./win32.h:314:33: error: 'FILE' {aka 'struct _iobuf'} has no member named '_file'
.... more missing member reports followed


Cheers,
Rob