`CultureNotFoundException` thrown when a keyboard with language tag `und` is activated

[mcdurdin]

Description

If a user installs and activates an input method with the custom locale und-Latn, a .NET application will instantly crash with CultureNotFoundException thrown ultimately by System.Globalization.CultureData.GetCultureData(Int32, Boolean).

Reproduction Steps

1. Install the custom language und-Latn using e.g. PowerShell:

   $1 = Get-WinUserLanguageList
   $1.Add("und-Latn")
   Set-WinUserLanguageList $1 -Force

2. Create a new WPF Application in Visual Studio from the New Project Wizard, targeting .NET 7.0 or later.
3. Add a TextBox to MainWindow.xaml:

   <TextBox x:Name="textBox" HorizontalAlignment="Left" Margin="138,123,0,0" TextWrapping="Wrap" Text="TextBox" VerticalAlignment="Top" Width="120"/>

4. Run the application, focus the text field.
5. Select the UND language from the Windows language picker.
6. The app will instantly crash.

Expected behavior

The language should be selected and the app should not crash.

Actual behavior

The application crashes with the exception CultureNotFoundException

Call stack from a crash dump given me by a user experiencing this problem:

0:000> !CLRStack -a
OS Thread Id: 0x7f88 (0)
        Child SP               IP Call Site
000000FD163ACBF8 00007ffc2c36fec4 [HelperMethodFrame: 000000fd163acbf8] 
000000FD163ACCF0 00007ffb62b5ec5b System.Globalization.CultureData.GetCultureData(Int32, Boolean)
    PARAMETERS:
000000FD163ACD30 00007ffb62b64f41 System.Globalization.CultureInfo..ctor(Int32, Boolean)
    PARAMETERS:
        this (<CLR reg>) = 0x000002ab8e7dda08
000000FD163ACD70 00007ffb0ee515f1 System.Windows.Input.InputLanguageSource.get_CurrentInputLanguage()
000000FD163ACDB0 00007ffb0ee51538 System.Windows.Input.InputLanguageManager.get_CurrentInputLanguage()
000000FD163ACE00 00007ffb0ee50fb2 System.Windows.Documents.TextSelection.EnsureCaret()
000000FD163ACE50 00007ffb100bf2d8 System.Windows.Documents.TextSelection.System.Windows.Documents.ITextSelection.UpdateCaretAndHighlight()
000000FD163ACEB0 00007ffb0ee46fe9 System.Windows.Documents.TextEditor.OnGotKeyboardFocus()
000000FD163ACF00 00007ffb0ee46e74 System.Windows.Documents.TextEditor.OnGotKeyboardFocus()
000000FD163ACF30 00007ffb0ee46e2b System.Windows.Controls.Primitives.TextBoxBase.OnGotKeyboardFocus()
000000FD163ACF60 00007ffb0f5b2237 System.Windows.RoutedEventArgs.InvokeHandler()
000000FD163ACFA0 00007ffb0f595359 System.Windows.EventRoute.InvokeHandlersImpl()
000000FD163AD090 00007ffb0f594853 System.Windows.UIElement.RaiseEventImpl()
000000FD163AD100 00007ffb0f616a38 System.Windows.UIElement.RaiseTrustedEvent()
000000FD163AD140 00007ffb0f643a5c System.Windows.Input.InputManager.ProcessStagingArea()
000000FD163AD220 00007ffb100ad87f System.Windows.Input.KeyboardDevice.ChangeFocus()
000000FD163AD2B0 00007ffb100463e9 System.Windows.Input.KeyboardDevice.Focus()
000000FD163AD320 00007ffb10046241 System.Windows.Input.KeyboardDevice.Focus()
000000FD163AD370 00007ffb10093c9e System.Windows.UIElement.Focus()
000000FD163AD3B0 00007ffb1065af1d Libronix.Utility.Windows.TextBoxUtility.AutoSelectAllTextBox_PreviewMouseDown()
000000FD163AD410 00007ffb0f5b2237 System.Windows.RoutedEventArgs.InvokeHandler()
000000FD163AD450 00007ffb0f595725 System.Windows.EventRoute.InvokeHandlersImpl()
000000FD163AD540 00007ffb0f594853 System.Windows.UIElement.RaiseEventImpl()
000000FD163AD5B0 00007ffb0f616a38 System.Windows.UIElement.RaiseTrustedEvent()
000000FD163AD5F0 00007ffb0f643a5c System.Windows.Input.InputManager.ProcessStagingArea()
000000FD163AD6D0 00007ffb0f8f742a System.Windows.Input.InputProviderSite.ReportInput()
000000FD163AD720 00007ffb0f917e65 System.Windows.Interop.HwndMouseInputProvider.ReportInput()
000000FD163AD820 00007ffb0f629999 System.Windows.Interop.HwndMouseInputProvider.FilterMessage()
000000FD163AD8C0 00007ffb0f6295ed System.Windows.Interop.HwndSource.InputFilterMessage()
000000FD163AD930 00007ffb0efa5919 MS.Win32.HwndWrapper.WndProc()
000000FD163AD9C0 00007ffb0efa5837 MS.Win32.HwndSubclass.DispatcherCallbackOperation()
000000FD163ADA10 00007ffb0cac4d43 System.Windows.Threading.ExceptionWrapper.InternalRealCall()
000000FD163ADA70 00007ffb0cac4b52 System.Windows.Threading.ExceptionWrapper.TryCatchWhen()
000000FD163ADAC0 00007ffb0ef78f4f System.Windows.Threading.Dispatcher.LegacyInvokeImpl()
000000FD163ADB50 00007ffb0efa5602 MS.Win32.HwndSubclass.SubclassWndProc()
000000FD163ADC40 00007ffb09b83ea0 ILStubClass.IL_STUB_ReversePInvoke(Int64, Int32, Int64, Int64)
    PARAMETERS:
000000FD163ADEC8 00007ffc2ab98241 [InlinedCallFrame: 000000fd163adec8] MS.Win32.UnsafeNativeMethods.DispatchMessage()
000000FD163ADEC8 00007ffb0bbf9ffb [InlinedCallFrame: 000000fd163adec8] MS.Win32.UnsafeNativeMethods.DispatchMessage()
000000FD163ADEA0 00007ffb0bbf9ffb ILStubClass.IL_STUB_PInvoke
    PARAMETERS:
000000FD163ADF60 00007ffb10080564 System.Windows.Threading.Dispatcher.PushFrameImpl()
000000FD163AE080 00007ffb0b1d7149 System.Windows.Threading.Dispatcher.PushFrame()
000000FD163AE100 00007ffb0b1d7277 System.Windows.Threading.Dispatcher.Run()
000000FD163AE140 00007ffb0b1d73b7 System.Windows.Application.RunDispatcher()
000000FD163AE180 00007ffb0b1d7b44 System.Windows.Application.RunInternal()
000000FD163AE270 00007ffb0b1d7baa System.Windows.Application.Run()
000000FD163AE2A0 00007ffb0b1d7bfa System.Windows.Application.Run()
000000FD163AE2D0 00007ffb09b8b715 LDLS4.OurApp.InitializeAndRun()
000000FD163AE3C0 00007ffb09b2fedf LDLS4.OurApp.Main()

Regression?

Unclear, although it appears that 4.x was working okay when I attempted a repro there.

Known Workarounds

Don't use the und language tag.

Configuration

• .NET Framework 7.0.10+a6dbb800a47735bde43187350fd3aff4071c7f9c (7.0.1023.36312)
• Windows 10 Pro 22H2 and others
• x64

Other information

It appears that this is related to the use of the ICU API uloc_canonicalize() in pal_locale.c:GetLocale(), which treats und as a default or "no language" and strips it off, returning _Latn when passed und-Latn.

FixupLocaleName() in pal_locale.c will convert _ back to - to bring the tag back into the format expected by .NET framework. However, the edge case of und being removed entirely is not handled. It may be possible to replace FixupLocaleName() with a call to uloc_toLanguageTag(): uloc_toLanguageTag("_Latn") yields und-Latn (but I have not verified that this is an exact match for requirements).

ICU bug reports:

• https://unicode-org.atlassian.net/browse/ICU-12338 is related and discusses some of this behaviour in uloc_canonicalize().

Keyman bug report -- original report:

• bug(windows): SIL IPA keyboard can cause .NET applications to crash keymanapp/keyman#10727

Other issues that may be related:

• CultureNotFoundException in System.Globalization.CultureData.GetCultureData #37789 (related to transient LCIDs rather than und)
• [8.0.100-preview.6.23310.1] System.Globalization.CultureNotFoundException: "Culture is not supported." #87394 (loosely related, re format of language tags)

Many thanks to @srl295 for his assistance in tracing this.

Tagging subscribers to this area: @dotnet/area-system-globalization
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Description

If a user installs and activates an input method with the custom locale und-Latn, a .NET application will instantly crash with CultureNotFoundException thrown ultimately by System.Globalization.CultureData.GetCultureData(Int32, Boolean).

Reproduction Steps

1. Install the custom language und-Latn using e.g. PowerShell:

   $1 = Get-WinUserLanguageList
   $1.Add("und-Latn")
   Set-WinUserLanguageList $1 -Force

2. Create a new WPF Application in Visual Studio from the New Project Wizard, targeting .NET 7.0 or later.
3. Add a TextBox to MainWindow.xaml:

   <TextBox x:Name="textBox" HorizontalAlignment="Left" Margin="138,123,0,0" TextWrapping="Wrap" Text="TextBox" VerticalAlignment="Top" Width="120"/>

4. Run the application, focus the text field.
5. Select the UND language from the Windows language picker.
6. The app will instantly crash.

Expected behavior

The language should be selected and the app should not crash.

Actual behavior

The application crashes with the exception CultureNotFoundException

Call stack from a crash dump given me by a user experiencing this problem:

0:000> !CLRStack -a
OS Thread Id: 0x7f88 (0)
        Child SP               IP Call Site
000000FD163ACBF8 00007ffc2c36fec4 [HelperMethodFrame: 000000fd163acbf8] 
000000FD163ACCF0 00007ffb62b5ec5b System.Globalization.CultureData.GetCultureData(Int32, Boolean)
    PARAMETERS:
000000FD163ACD30 00007ffb62b64f41 System.Globalization.CultureInfo..ctor(Int32, Boolean)
    PARAMETERS:
        this (<CLR reg>) = 0x000002ab8e7dda08
000000FD163ACD70 00007ffb0ee515f1 System.Windows.Input.InputLanguageSource.get_CurrentInputLanguage()
000000FD163ACDB0 00007ffb0ee51538 System.Windows.Input.InputLanguageManager.get_CurrentInputLanguage()
000000FD163ACE00 00007ffb0ee50fb2 System.Windows.Documents.TextSelection.EnsureCaret()
000000FD163ACE50 00007ffb100bf2d8 System.Windows.Documents.TextSelection.System.Windows.Documents.ITextSelection.UpdateCaretAndHighlight()
000000FD163ACEB0 00007ffb0ee46fe9 System.Windows.Documents.TextEditor.OnGotKeyboardFocus()
000000FD163ACF00 00007ffb0ee46e74 System.Windows.Documents.TextEditor.OnGotKeyboardFocus()
000000FD163ACF30 00007ffb0ee46e2b System.Windows.Controls.Primitives.TextBoxBase.OnGotKeyboardFocus()
000000FD163ACF60 00007ffb0f5b2237 System.Windows.RoutedEventArgs.InvokeHandler()
000000FD163ACFA0 00007ffb0f595359 System.Windows.EventRoute.InvokeHandlersImpl()
000000FD163AD090 00007ffb0f594853 System.Windows.UIElement.RaiseEventImpl()
000000FD163AD100 00007ffb0f616a38 System.Windows.UIElement.RaiseTrustedEvent()
000000FD163AD140 00007ffb0f643a5c System.Windows.Input.InputManager.ProcessStagingArea()
000000FD163AD220 00007ffb100ad87f System.Windows.Input.KeyboardDevice.ChangeFocus()
000000FD163AD2B0 00007ffb100463e9 System.Windows.Input.KeyboardDevice.Focus()
000000FD163AD320 00007ffb10046241 System.Windows.Input.KeyboardDevice.Focus()
000000FD163AD370 00007ffb10093c9e System.Windows.UIElement.Focus()
000000FD163AD3B0 00007ffb1065af1d Libronix.Utility.Windows.TextBoxUtility.AutoSelectAllTextBox_PreviewMouseDown()
000000FD163AD410 00007ffb0f5b2237 System.Windows.RoutedEventArgs.InvokeHandler()
000000FD163AD450 00007ffb0f595725 System.Windows.EventRoute.InvokeHandlersImpl()
000000FD163AD540 00007ffb0f594853 System.Windows.UIElement.RaiseEventImpl()
000000FD163AD5B0 00007ffb0f616a38 System.Windows.UIElement.RaiseTrustedEvent()
000000FD163AD5F0 00007ffb0f643a5c System.Windows.Input.InputManager.ProcessStagingArea()
000000FD163AD6D0 00007ffb0f8f742a System.Windows.Input.InputProviderSite.ReportInput()
000000FD163AD720 00007ffb0f917e65 System.Windows.Interop.HwndMouseInputProvider.ReportInput()
000000FD163AD820 00007ffb0f629999 System.Windows.Interop.HwndMouseInputProvider.FilterMessage()
000000FD163AD8C0 00007ffb0f6295ed System.Windows.Interop.HwndSource.InputFilterMessage()
000000FD163AD930 00007ffb0efa5919 MS.Win32.HwndWrapper.WndProc()
000000FD163AD9C0 00007ffb0efa5837 MS.Win32.HwndSubclass.DispatcherCallbackOperation()
000000FD163ADA10 00007ffb0cac4d43 System.Windows.Threading.ExceptionWrapper.InternalRealCall()
000000FD163ADA70 00007ffb0cac4b52 System.Windows.Threading.ExceptionWrapper.TryCatchWhen()
000000FD163ADAC0 00007ffb0ef78f4f System.Windows.Threading.Dispatcher.LegacyInvokeImpl()
000000FD163ADB50 00007ffb0efa5602 MS.Win32.HwndSubclass.SubclassWndProc()
000000FD163ADC40 00007ffb09b83ea0 ILStubClass.IL_STUB_ReversePInvoke(Int64, Int32, Int64, Int64)
    PARAMETERS:
000000FD163ADEC8 00007ffc2ab98241 [InlinedCallFrame: 000000fd163adec8] MS.Win32.UnsafeNativeMethods.DispatchMessage()
000000FD163ADEC8 00007ffb0bbf9ffb [InlinedCallFrame: 000000fd163adec8] MS.Win32.UnsafeNativeMethods.DispatchMessage()
000000FD163ADEA0 00007ffb0bbf9ffb ILStubClass.IL_STUB_PInvoke
    PARAMETERS:
000000FD163ADF60 00007ffb10080564 System.Windows.Threading.Dispatcher.PushFrameImpl()
000000FD163AE080 00007ffb0b1d7149 System.Windows.Threading.Dispatcher.PushFrame()
000000FD163AE100 00007ffb0b1d7277 System.Windows.Threading.Dispatcher.Run()
000000FD163AE140 00007ffb0b1d73b7 System.Windows.Application.RunDispatcher()
000000FD163AE180 00007ffb0b1d7b44 System.Windows.Application.RunInternal()
000000FD163AE270 00007ffb0b1d7baa System.Windows.Application.Run()
000000FD163AE2A0 00007ffb0b1d7bfa System.Windows.Application.Run()
000000FD163AE2D0 00007ffb09b8b715 LDLS4.OurApp.InitializeAndRun()
000000FD163AE3C0 00007ffb09b2fedf LDLS4.OurApp.Main()

Regression?

Unclear, although it appears that 4.x was working okay when I attempted a repro there.

Known Workarounds

Don't use the und language tag.

Configuration

• .NET Framework 7.0.10+a6dbb800a47735bde43187350fd3aff4071c7f9c (7.0.1023.36312)
• Windows 10 Pro 22H2 and others
• x64

Other information

It appears that this is related to the use of the ICU API uloc_canonicalize() in pal_locale.c:GetLocale(), which treats und as a default or "no language" and strips it off, returning _Latn when passed und-Latn.

FixupLocaleName() in pal_locale.c will convert _ back to - to bring the tag back into the format expected by .NET framework. However, the edge case of und being removed entirely is not handled. It may be possible to replace FixupLocaleName() with a call to uloc_toLanguageTag(): uloc_toLanguageTag("_Latn") yields und-Latn (but I have not verified that this is an exact match for requirements).

ICU bug reports:

• https://unicode-org.atlassian.net/browse/ICU-12338 is related and discusses some of this behaviour in uloc_canonicalize().

Keyman bug report -- original report:

• bug(windows): SIL IPA keyboard can cause .NET applications to crash keymanapp/keyman#10727

Other issues that may be related:

• CultureNotFoundException in System.Globalization.CultureData.GetCultureData #37789 (related to transient LCIDs rather than und)
• [8.0.100-preview.6.23310.1] System.Globalization.CultureNotFoundException: "Culture is not supported." #87394 (loosely related, re format of language tags)

Author:	mcdurdin
Assignees:	-
Labels:	area-System.Globalization, untriaged, needs-area-label
Milestone:	-

[tarekgh]

I think there is some confusion here. The thrown exception is not because .NET couldn't handle the name und-Latn but because WPF is trying to create the culture using invalid LCID. Look at the stack and you will see System.Globalization.CultureInfo..ctor(Int32, Boolean). If you have the full exception message, it should tell what LCID value was used. My guess here is WPF trying to get LCID of the input keyboard from Windows and then try to create the culture using that LCID. It is possible the LCID returned from Windows is 0x1000 which is a custom LCID which cannot create a culture using such LCID.

This doesn't mean .NET will succeed creating a culture with the name und-Latn, that is because we use ICU API uloc_getName which truncate the language part. We already discussed this issue with the ICU guys in the ticket https://unicode-org.atlassian.net/browse/ICU-22137 and we are considering switch using uloc_toLanguageTag instead and ensure canonicalize the culture name too. I'll keep this issue open to track that work.

[mcdurdin]

@tarekgh that was my initial assumption too (see #37789 which was precisely that). However, when I debugged my sample application, I found that .NET is translating from the transient locale 0x2000 correctly to get und-Latn, which it then passes in to ICU.

To prove this, I added another transient locale, this time picking tpi-PG (Tok Pisin, Papua New Guinea), which was assigned LCID 0x2400:

PS C:\Users\mcdurdin> $1 = Get-WinUserLanguageList
PS C:\Users\mcdurdin> $1.add("tpi-pg")
PS C:\Users\mcdurdin> Set-WinUserLanguageList $1 -force
PS C:\Users\mcdurdin> Get-WinUserLanguageList


LanguageTag     : en-AU
Autonym         : English (Australia)
EnglishName     : English
LocalizedName   : English (Australia)
ScriptName      : Latin
InputMethodTips : {0C09:00000409}
Spellchecking   : True
Handwriting     : False

LanguageTag     : km
Autonym         : ភាសាខ្មែរ
EnglishName     : Khmer
LocalizedName   : Khmer
ScriptName      : Khmer
InputMethodTips : {0453:{FE0420F1-38D1-4B4C-96BF-E7E20A74CFB7}{99A91333-C033-428B-A076-D44C9FE0F450}}
Spellchecking   : True
Handwriting     : False

LanguageTag     : und-Latn
Autonym         :
EnglishName     : Undetermined (und-Latn)
LocalizedName   :
ScriptName      : Latin
InputMethodTips : {2000:00000409}
Spellchecking   : True
Handwriting     : False

LanguageTag     : tpi-PG
Autonym         : Tok Pisin (tpi-PG)
EnglishName     : Tok Pisin (tpi-PG)
LocalizedName   : Tok Pisin (tpi-PG)
ScriptName      : Latin
InputMethodTips : {2400:00000409}
Spellchecking   : True
Handwriting     : False

Selecting and using tpi-PG never caused the exception, whereas selecting und-Latn consistently did (note: if the text field was already focused when I switched languages, then the first keystroke would throw).

I debugged GetCultureData(int,bool) in CultureData.cs:

            // Convert the lcid to a name, then use that
            string? localeName = LCIDToLocaleName(culture);

            if (!string.IsNullOrEmpty(localeName))
            {
                // Valid name, use it
                retVal = GetCultureData(localeName, bUseUserOverride);
            }

LCIDToLocaleName is passed 0x2000 and returns und-Latn. GetCultureData('und-Latn',true) returns null.

• Parameter bUseUserOverride == true per the call from System.Windows.Input.InputLanguageSource.get_CurrentInputLanguage().

GlobalizationMode flags are all default and recommended settings:

• GlobalizationMode.PredefinedCulturesOnly == false
• GlobalizationMode.Invariant == false
• GlobalizationMode.UseNls == false

Rough sequence of events:

1. GetCultureData(string,bool) calls CreateCultureData(cultureName, useUserOverride) (because the language is not yet in the cached cultures hash table)
2. That then calls CultureData.InitCultureDataCore().
3. Which calls InitIcuCultureDataCore().
4. InitIcuCultureDataCore() ends up with _sRealName set to "-Latn" as a result of the call to GetLocaleName() in the ICU wrapper pal_locale.c.
5. InitCultureDataCore() then passes this bogus string to Windows API GetLocaleInfoEx() (which of course balks at that).

So I think my initial analysis is correct.

Side note: as an experiment, in the debugger, I overrode the _sWindowsName value returned from GetLocaleName() and set it back to 'und-Latn'. That avoided the crash -- the culture data was successfully initialized and added to the cached culture hash table. No other issues were encountered.

[tarekgh]

@mcdurdin thanks for your investigation and analysis. I agree with you now 😄. I didn't know that WPF was getting a transient LCID 0x2000 and passing it to the .NET. This explains why this worked on the .NET Framework too.

By the way, you can enable Using NLS instead of ICU which should make it work without hacking values under the debugger.

[mcdurdin]

Thanks-- I will document the globalization nls flag as a workaround for our users as well. It was useful to verify the root cause in the debugger though without other potential side effects 😀

[srl295]

we are considering switch using uloc_toLanguageTag instead and ensure canonicalize the culture name too. I'll keep this issue to track that work.

If you want bcp47 you should definitely be using uloc_toLanguageTag. We wrote the other apis long before bcp47 existed, now they should have warnings on them.