feat(ui): Enhances virtual keyboard with sticky modifier key support #500

IDisposable · 2025-05-22T10:25:57Z

Adds support for Shift, Ctrl, Alt, Meta, and AltGr keys to the virtual keyboard, treating them as "sticky" keys when entered on the Virtual Keyboard. This allows the user to click the Shift or Ctrl or Alt or Meta or AltGr button and then click another button to emit the combo (e.g. Ctrl-Alt-Del would be done using exactly that click sequence).

Tracks and displays the active state of modifier keys (Shift, Ctrl, Alt, Meta, AltGr, CapsLock, NumLock, ScrollLock) in the InfoBar.
Updates the virtual keyboard to reflect the state of these modifier keys, including a "depressed" visual style.
Changes the virtual keyboard layout based on current Shift and CapsLock states.
Improves keyboard event handling to correctly send keycodes with appropriate modifiers.
Updated the InfoBar to show the status of all sticky keys
Added code to the WebRTCVideo.tsx to ensure that physical keys are also kept in sync so you can use have the Virtual Keyboard up, use the physical Ctrl (or other) key and click the desired character. For example Ctrl-C

IDisposable · 2025-05-22T10:30:54Z

@ym @adamshiervani This is ready for review and works really well ;)

I am not sure about the CSS style of the "depressed" buttons, I made something up and welcome other suggestions.

IDisposable · 2025-05-22T10:33:05Z

The sticky keys and InfoBar look like this

IDisposable · 2025-05-22T10:34:17Z

ui/src/components/InfoBar.tsx

@@ -37,6 +37,11 @@ export default function InfoBar() {
  }, [rpcDataChannel]);

  const isCapsLockActive = useHidStore(state => state.isCapsLockActive);
+  const isShiftActive = useHidStore(state => state.isShiftActive);


State us kept in sync between the physical keyboard (in WebRTCVideo.tsx) and the virtual keyboard (in VirtualKeyboard.tsx)

IDisposable · 2025-05-22T10:35:09Z

ui/src/components/InfoBar.tsx

@@ -118,6 +123,56 @@ export default function InfoBar() {
              Relayed by Cloudflare
            </div>
          )}
+          <div


We might want to hide some of these if the browser window get smaller eventually.

IDisposable · 2025-05-22T10:36:45Z

ui/src/components/VirtualKeyboard.tsx

@@ -26,6 +26,7 @@ const AttachIcon = ({ className }: { className?: string }) => {

 function KeyboardWrapper() {
  const [layoutName, setLayoutName] = useState("default");
+  const [depressedButtons, setDepressedButtons] = useState("");


This tracks which buttons are "sticky down" to add them to the main keyboard's depressed-key class... so they get a different CSS style.

IDisposable · 2025-05-22T10:37:31Z

ui/src/components/VirtualKeyboard.tsx

+  useEffect(() => {
+    // if you have the CapsLock "down", then the shift state is inverted
+    const effectiveShift = isCapsLockActive ? false === isShiftActive : isShiftActive;
+    setLayoutName(effectiveShift ? "shift" : "default");


Using a useEffect makes this so much easier... and it stays in sync :)

IDisposable · 2025-05-22T10:38:02Z

ui/src/components/VirtualKeyboard.tsx

-          [keys["Escape"]],
-          [modifiers["MetaLeft"], modifiers["AltLeft"]],
-        );
+  // this causes the buttons to look depressed/clicked depending on the sticky state


This is where we build up the button names of the ones that should be stuck down.

IDisposable · 2025-05-22T10:39:28Z

ui/src/components/VirtualKeyboard.tsx

+    const cleanKey = key.replace(/[()]/g, "");
+
+    // if it's one of the special keys, just pass it through immediately and return
+    const passthrough = ["PrintScreen", "SystemRequest", "Pause", "Break", "ScrollLock", "Enter", "Space"].find((value) => value === cleanKey);


For the keys that should just get emitted without further massaging, we have a preemptive emit and return

IDisposable · 2025-05-22T10:40:11Z

ui/src/components/VirtualKeyboard.tsx

+      return;
+    }
+
+    // adjust the sticky state of the Shift/Ctrl/Alt/Meta/AltGr


Here's where we take the key click and turn it into toggling the modifier states.

IDisposable · 2025-05-22T10:41:04Z

ui/src/components/VirtualKeyboard.tsx

+    emitkeycode(cleanKey);
+
+    function emitkeycode(key: string) {
+      const effectiveMods: number[] = [];


And now, since we're tracking HID wide, the state of the modifiers we can just compute it!

IDisposable · 2025-05-22T10:42:46Z

ui/src/components/VirtualKeyboard.tsx

+        sendKeyboardEvent([keycode], effectiveMods);
+      }
+
+      // release the key (if one pressed), but retain the modifiers


Here's the biggest gain... we can leave the modifier state still reflecting what we're tracking it to be so that the remote knows we still have the corresponding keys down. This is especially important for CapsLock because we want the remote to know we've got that on so it can do the warnings and such.

IDisposable · 2025-05-22T10:43:11Z

ui/src/components/VirtualKeyboard.tsx

                        display={keyDisplayMap}
                        layout={{
-                          default: ["PrintScreen ScrollLock Pause", "Insert Home Pageup", "Delete End Pagedown"],
-                          shift: ["(PrintScreen) ScrollLock (Pause)", "Insert Home Pageup", "Delete End Pagedown"],
+                          default: ["PrintScreen ScrollLock Pause", "Insert Home PageUp", "Delete End PageDown"],


Oops, fixed the name

IDisposable · 2025-05-22T10:43:45Z

ui/src/components/WebRTCVideo.tsx

@@ -56,8 +56,16 @@ export default function WebRTCVideo() {
  const isVideoLoading = !isPlaying;

  // Keyboard related states
-  const { setIsNumLockActive, setIsCapsLockActive, setIsScrollLockActive } =
-    useHidStore();
+  const { 


Here we're tracking the physical keyboard state to keep things in sync

IDisposable · 2025-05-22T10:44:26Z

ui/src/index.css

@@ -222,6 +222,10 @@ video::-webkit-media-controls {
  background: none;
 }

+.simple-keyboard .hg-button.depressed-key {


This is the "depressed button" style... I welcome better ideas.

IDisposable · 2025-05-22T10:45:43Z

ui/src/keyboardMappings.ts

-  Tab: "tab",
-  Backspace: "backspace",
-  "(Backspace)": "backspace",
+  Tab: "tab ⇥",


Added the cool unicode characters for the keys, so the keyboard looks nicer now.

It's actually on keyboard as (Pause)

Ensures the InfoBar tracks for physical and virtual keyboard

Now treats all the Shift/Control/Alt/Meta/AltGr keys as if they were sticky keys so users can click the button and hit the next key,

IDisposable · 2025-05-23T08:13:23Z

Rebased and ready for review @ym

IDisposable · 2025-05-23T08:15:35Z

ui/src/components/VirtualKeyboard.tsx

+    }
+
+    // adjust the sticky state of the Shift/Ctrl/Alt/Meta/AltGr
+    if (key === "CapsLock" && !isKeyboardLedManagedByHost)


Note that this does integrate the new server-side test https://github.com/jetkvm/kvm/pull/500/files#diff-629810bc8df4bc99558ef987d2c89750fa7f782dfc87b9302c5e66e51f9031ebL180

IDisposable commented May 22, 2025

View reviewed changes

IDisposable marked this pull request as ready for review May 22, 2025 10:45

IDisposable changed the title ~~Enhances virtual keyboard with sticky modifier key support~~ feat(ui): Enhances virtual keyboard with sticky modifier key support May 22, 2025

IDisposable mentioned this pull request May 22, 2025

Virtual keyboard is missing SysRq/PrintScreen #77

Open

IDisposable added 8 commits May 23, 2025 02:38

Made the operation keys look more like a keyboard

4e9775d

Add persistent state management for the Shift/Ctrl/Alt/Meta

fee563e

Add infoBar indicators for Shift/Ctr/Alt/Meta/AltGr

10a0526

Remove redundant Break

db3eaf9

It's actually on keyboard as (Pause)

Add a style for the virtual keyboard "depressed" buttons.

5325f9c

Delete the no-longer-needed button variants

15ee224

Added missing keycodes

7d15de3

Added modifier tracking for all the other keys

d1fd1a5

Ensures the InfoBar tracks for physical and virtual keyboard

IDisposable force-pushed the feat/virtual-keyboard-sticky branch from 5be305a to 2020b6c Compare May 23, 2025 08:06

Shows what buttons are depressed for sticky keys

642f0a5

Now treats all the Shift/Control/Alt/Meta/AltGr keys as if they were sticky keys so users can click the button and hit the next key,

IDisposable force-pushed the feat/virtual-keyboard-sticky branch from 2020b6c to 642f0a5 Compare May 23, 2025 08:12

IDisposable commented May 23, 2025

View reviewed changes

feat(ui): Enhances virtual keyboard with sticky modifier key support #500

Are you sure you want to change the base?

feat(ui): Enhances virtual keyboard with sticky modifier key support #500

Conversation

IDisposable commented May 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

IDisposable commented May 22, 2025

Uh oh!

IDisposable commented May 22, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

IDisposable commented May 23, 2025

Uh oh!

IDisposable May 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

IDisposable commented May 22, 2025 •

edited

Loading

IDisposable May 23, 2025 •

edited

Loading