Fatal error with catacharset.cpp that cause the game to crash

[TheCYPER]

Describe the bug

I'm tried the experimental versions experimental build 2023-12-17-0336, experimental build 2023-12-18-2119, experimental build 2023-12-20-0338, and experimental build 2023-12-20-2343.
But All the game crashed for the same error in Macos crash report

Application Specific Information:
Assertion failed: (sz != static_cast<size_t>( -1 )), function utf8_to_wstr, file catacharset.cpp, line 350.

Attach save file

拉斯维加斯-trimmed.tar.gz

Steps to reproduce

1. I'm not sure if there is a clear reproduce step because its just randomly happening
2. try moving to southern east for some steps, and then south
3. If not happening, maybe try to play around for a bit, it might be related to time
4. DEBUG : error: item condom cannot fit into pocket while loading: 物品太长

FUNCTION : void item_contents::combine(const item_contents &, const bool, const bool, bool, bool)
FILE : src/item_contents.cpp
LINE : 706
VERSION : a4a6785
This Debug message is usually happened before or after the game crashes for several times
But I don't have any condom in my pocket

Expected behavior

(The game not crashing?)

Screenshots

No response

Versions and configuration

• OS: MacOs
  • OS Version: Name: macOS; Version: 11.7.4; Build: 20G1120;
• Game Version: a4a6785 [64-bit]
• Graphics Version: Tiles
• Game Language: 中文 (天朝)<color_dark_gray>（88.5%） [zh_CN]
• Mods loaded: [
  浩劫余生 [dda],
  NPC无基本需求 [no_npc_food],
  NPC不受空间风暴影响 [personal_portal_storms],
  生化职业 [package_bionic_professions],
  大魔法 [magiclysm],
  Blaze 载具工业 [blazeindustries],
  XEDRA 超进化 [xedra_evolved],
  翻译晦涩对话 [translate_dialogue],
  疯狂大灾变 [crazy_cataclysm],
  更多变异场景 [extra_mut_scens],
  神话武术 [MMA],
  技能提升属性 [StatsThroughSkills],
  敏捷速度加成 [speedydex],
  杀怪提升属性 [stats_through_kills],
  驯服野兽 [Tamable_Wildlife],
  浮空岛 [skyisland](I think this one caused the crash?),
  心胜于物 [mindovermatter],
  巨兽 [megafauna],
  恐龙世界 [DinoMod],
  特色升级能力 [bombastic_perks],
  减缓真菌增殖 [no_fungal_growth]
  ]

Additional context

This is my config.zip
config.zip
This is my Crash report in MacOS
Crash report in OS.txt

[TheCYPER]

Steps to reproduce
try moving to southern east for some steps, and then east
I made a mistake here

[ZhilkinSerg]

See #68558. You might need to change your locale settings.

[TheCYPER]

here are some things i tested in these days

1. it is not a problem with #68558 after i did the things in Locale. it is still crashing. Entering undamaged car seems to be also unrelated with this problem
2. Enter in a city is a very important factor with this problem. When entering the city after getting off the island for a long time, it's easy to start to have this issue
3. In all the new version, this problem is not solved

What are some other thing I could submit to help developing? And I will go looking at line 350 in catacharset.cpp

[TheCYPER]

mac@PercydeMacBook-Air ~> locale
LANG="zh_CN.UTF-8"
LC_COLLATE="zh_CN.UTF-8"
LC_CTYPE="zh_CN.UTF-8"
LC_MESSAGES="zh_CN.UTF-8"
LC_MONETARY="zh_CN.UTF-8"
LC_NUMERIC="zh_CN.UTF-8"
LC_TIME="zh_CN.UTF-8"
LC_ALL=

This is my current locale setting on my mac and I'm still having this problem I mention.
I know is assertion fail related to locale setting since I see many guys having the same issue.
#72500
#73003
#68558
And they are all Unix-like systems and use English as their main language

However, after I see 350 in catacharset.cpp

std::wstring utf8_to_wstr( const std::string &str )
{
#if defined(_WIN32)
    int sz = MultiByteToWideChar( CP_UTF8, 0, str.c_str(), -1, nullptr, 0 ) + 1;
    std::wstring wstr( sz, '\0' );
    MultiByteToWideChar( CP_UTF8, 0, str.c_str(), -1, wstr.data(), sz );
    strip_trailing_nulls( wstr );
    return wstr;
#else
    std::size_t sz = std::mbstowcs( nullptr, str.c_str(), 0 );     # Using mbstowcs to get sz
    cata_assert( sz != static_cast<size_t>( -1 ) );                    # this one, line 350
    std::wstring wstr( sz + 1, '\0' );
    [[maybe_unused]] const size_t converted = std::mbstowcs( wstr.data(), str.c_str(), sz );
    cata_assert( converted == sz );
    strip_trailing_nulls( wstr );
    return wstr;
#endif
}

This should be a problem with mbstowcs function returning a -1, which is a problem with C's function
Thus,

• How is this related to my own locale setting
• How could I change my settings to make things work on my osx, since many command is not avalible on osx and my main language is chinese

[duncanyoyo1]

I have been experiencing this crash on a few different systems. One of them is muOS which is a buildroot based distro, so linux kernel 4.9.170 and a very minimal environment. I have tried changing my locale settings, and all the relevant things are set as you would expect. E.g. LC_ALL, LC_CTYPE, LANG, etc.

I made a small change to the catacharset.cpp to throw an error instead of crash and to use mbsrtowcs instead of mbstowcs which should be "safer" . So far I have not observed any negative effects.

std::wstring utf8_to_wstr(const std::string &str)
{
#if defined(_WIN32)
    int sz = MultiByteToWideChar(CP_UTF8, 0, str.c_str(), -1, nullptr, 0);
    if (sz == 0) {
        debugmsg("Failed to convert string to wide string (UTF-8 to wchar). Input: %s", str.c_str());
        return L""; // Return an empty string on failure
    }
    std::wstring wstr(sz, L'\0');
    MultiByteToWideChar(CP_UTF8, 0, str.c_str(), -1, wstr.data(), sz);
    strip_trailing_nulls(wstr);
    return wstr;
#else
    std::mbstate_t state = std::mbstate_t(); // Initialize to zero
    const char *src = str.c_str();
    std::size_t len = std::mbsrtowcs(nullptr, &src, 0, &state);
    if (len == static_cast<size_t>(-1)) {
        debugmsg("Invalid UTF-8 sequence encountered. Input: %s", str.c_str());
        return L""; // Return an empty string on invalid input
    }
    std::wstring wstr(len, L'\0');
    std::mbsrtowcs(&wstr[0], &src, len, &state);
    strip_trailing_nulls(wstr);
    return wstr;
#endif
}

I did not dig further to see what is causing this, but the issue seems to be whatever is causing that function to return -1 to start with. Not necessarily the function itself.