Game mode IME: from no Chinese input to committed text showing as digits ("1"/"12") - root cause in keymap serialization

[huazhidajiji]

TL;DR

Full story of making Chinese (and any non-ASCII) IME input work in Steam game mode (gamescope session), from "cannot type Chinese at all" to "candidates show but committed text appears as digits 1 / 12" and finally to the root cause: gamescope's generated IME keymap loses its compatibility map when serialized with xkb_keymap_get_as_string(), so Xwayland's xkbcomp silently drops all symbols and the X server never applies the custom mapping.

Background (stage 1): no Chinese input in game mode at all

On a SteamOS-like handheld session (CachyOS, gamescope-session), the Steam virtual keyboard could not do Chinese input:

• fcitx5 could not connect to the gamescope Wayland socket (Failed to open wayland connection).
• Chinese candidates were provided through the IBus integration built into Steam's virtual keyboard (Steam keyboard JS acts as an IBus D-Bus client; ibus-daemon runs via ibus-gamescope.service).
• Getting candidates to show at all required: Steam client on the steamdeck preview channel, ibus + Chinese engine configured, and the game mode session environment fixed (see earlier work on ibus notifications / env propagation).

That part is configuration, not a gamescope bug. The gamescope-specific bug started at stage 2.

Stage 2: candidates show, but committing outputs digits "1" / "12"

After the setup worked, typing pinyin showed the Chinese candidate list inside the Steam keyboard, but selecting a candidate typed the digits 1 (one char) or 12 (two chars) into the text field instead of the Chinese text.

The commit path (confirmed from Steam keyboard JS chunk~2dcc5aaf7.js + protocol captures):

IBus commit "汉" -> Steam keyboard OnCommitText()
  -> VirtualKeyboardManager.HandleVirtualKeyDown("汉")
    -> DispatchKeypress -> SteamClient.Input.ControllerKeyboardSendText("汉")
      -> native Steam client -> gamescope_input_method.set_string("汉") + commit
        -> gamescope ime.cpp type_text():
             generate_keymap()  (keycode K2 -> keysym 0x1006c49 "汉")
             wlr_keyboard_set_keymap()  (serializes the keymap)
             inject fake keycode KEY_1 (=2) for 1st char, KEY_2 (=3) for 2nd
               -> Xwayland keycode 10 / 11 -> Steam CEF

KEY_1/KEY_2 are the first entries of allow_keycodes[] in ime.cpp, which is exactly why one char shows 1 and two chars show 12.

Stage 3: root cause — the keymap is rejected before it ever reaches the X server

After deep tracing (WAYLAND_DEBUG on Xwayland, capturing wl_keyboard.keymap events, grabbing the .xkm files xkbcomp produces, and recompiling the exact keymap text), the chain is:

1. generate_keymap() emits xkb_compatibility "(unnamed)" { include "complete" }; which expands into the full set of interpret entries, and wlr_keyboard_set_keymap() is called.
2. wlroots serializes the keymap with xkb_keymap_get_as_string(). xkbcommon's serializer (keymap-dump.c::write_compat) drops every interpret whose keysym does not match a key present in the symbols map (if (!si->required && drop_unused) continue;). The fake keymap only contains the committed characters plus action keys (BackSpace/Return/arrows/Delete), so no interpret survives. The serialized xkb_compatibility section ends up with only interpret.useModMapMods / interpret.repeat and zero interpret blocks.
3. Xwayland's keyboard_handle_keymap() compiles that text with XkbCompileKeymapFromString() -> xkbcomp -xkm. With an empty compatibility map, xkbcomp silently drops the whole symbols section (observed: "compatibility map not defined", and the compiled .xkm contains keycodes/types only). The X server keymap never gets key <K2> -> 汉.
4. The injected keycodes are therefore resolved with the standard keymap -> 1, 12.

Verified live:

• The keymap event gamescope sends to clients contains key <K2> { [ 0x1006c49 ] }; but its serialized compatibility section has zero interpret entries (dumped the exact wl_keyboard.keymap bytes).
• Recompiling that serialized text with xkbcomp -xkm reproduces the dropped symbols.
• A delay-based workaround (waiting for Xwayland to pick up the keymap before pressing keys) changes nothing — this is not a timing race, it is a serialization/compilation loss.

Minimal reproduction (no Steam needed)

1. Start nested gamescope (gamescope --nested-width 1280 --nested-height 800).
2. Connect a gamescope_input_method client, send set_string("汉") + commit.
3. Watch the Xwayland server keymap (xkbcomp -xkb :N -): keycode 10 stays 1; with the fix below it becomes U6C49 during the commit.

Suggested fix (verified end-to-end)

Keep at least one interpret alive through serialization by adding a key with a modifier map, so xkbcommon's FindInterpForKey marks it required. In generate_keymap():

--- a/src/ime.cpp
+++ b/src/ime.cpp
@@ -209,6 +209,14 @@ static struct xkb_keymap *generate_keymap(struct wlserver_input_method *ime)
 		uint32_t keycode = kv.second.keycode;
 		fprintf(f, "	<K%u> = %u;\n", keycode, keycode + keycode_offset);
 	}
+	// xkbcommon's keymap serializer (used by wlr_keyboard_set_keymap) only
+	// keeps symbol interpretations that match a key present in the map, and
+	// xkbcomp (invoked by Xwayland) rejects the compatibility section when
+	// it contains no interpret entries, silently dropping all symbols (so
+	// typed text showed up as the digits of the fake keycodes). Provide a
+	// key with a modifier map so at least the Num_Lock interpretation from
+	// "complete" is retained in the serialized keymap.
+	fprintf(f, "	<K%u> = %u;\n", KEY_NUMLOCK, KEY_NUMLOCK + keycode_offset);
 
 	// TODO: should we really be including "complete" here? squeekboard seems
 	// to get away with some other workarounds:
@@ -240,6 +248,13 @@ static struct xkb_keymap *generate_keymap(struct wlserver_input_method *ime)
 			return nullptr;
 		}
 	}
+	if (!generate_keymap_key(f, KEY_NUMLOCK, XKB_KEY_Num_Lock))
+	{
+		fclose(f);
+		free(str);
+		return nullptr;
+	}
+	fprintf(f, "	modifier_map Mod2 { <K%u> };\n", KEY_NUMLOCK);
 
 	fprintf(f,
 		"};\n"

After the fix, verified end-to-end in nested gamescope with the real Steam client:

IME commit "汉字" ->
X server keymap during commit: keycode 10 -> U6C49(汉), keycode 11 -> U5B57(字)
X client receives:               KEY keycode=10 keysym=0x1006c49 (U6C49)

An alternative fix would be to bypass wlr_keyboard_set_keymap's serialization and send the keymap text verbatim (with include "complete"), which also keeps the compatibility section intact.