This is a draft proposal for a function that pastes strings into the guest by using the V86 keyboard with support for non-US keyboard layouts. This function is needed in different contexts:
- to paste a plaintext string from the system clipboard into V86
- to handle keyboard input in mobile browser environments
- in internal tests and other script-based V86 use cases
This function is currently supported through V86.keyboard_send_text(string, delay), but limited to the US keyboard layout.
Desktop browser keyboard input
Let's start with a deep look at how V86 handles keyboard input in a desktop browser environment and with some needed terms and concepts. A simplified view of the keyboard pipeline when V86 is used with a desktop browser can be depicted as:
DesktopKeyboard --> Scancode --> IRQ-1 --> ... --> GuestKeyboard
DesktopKeyboard represents the system keyboard state (the set of pressed keys), and GuestKeyboard the keyboard state as seen by the guest (GuestKeyboard is only a conceptual element which we don't implement). Our main task is to assert that the guest's keyboard state is always in sync with the system's keyboard state (except for key combinations that are intercepted by the OS or browser like Ctrl+Alt+Del).
A scancode is an 8- or 16-bit integer that represents the physical location of a key on a keyboard, a value that isn't altered by the visual keyboard layout or the state of the modifier keys. The 8th bit (0x80) of scancodes is reserved, when cleared it signals a "keydown" event, else a "keyup" event (these values are also called "make" and "break" codes, respectively).
Whenever the user presses or releases a key on the system keyboard its scancode and state is sent immediately to the V86 PS/2 controller and passed on to the guest's keyboard driver via interrupt. 16-bit scancodes are sent to the PS/2 controller as two separate bytes.
The desktop browser's KeyboardEvent.code property represents the same physical key location as a scancode but uses strings instead of integers to identify physical keys. "keydown" and "keyup" events are delivered in separate KeyboardEvent instances.
A 1:1 mapping between KeyboardEvent.code and scancodes exists which gives a well-defined and universal solution for this, independent of the visual keyboard layout.
This is how V86 handles this case, though there is also a fallback to KeyboardEvent.keyCode which only works for US-keyboards and has been deprecated in the standards 4 years ago or so, unless there are reasons to keep it I believe this fallback can be removed.
String-based keyboard input
Even though the keyboard pipeline can be used to "paste" plaintext strings into the guest, this method works very differently than the one described above. The pipeline looks like this while a string paste operation is in progress:
DesktopKeyboard --X
PasteKeyboard ----> Scancode --> IRQ-1 --> ... --> GuestKeyboard
DesktopKeyboard is muted here, meaning it still receives system keyboard events and updates its internal state but no scancodes are fed into the keyboard pipeline, the pipeline is instead fed by PasteKeyboard for the duration of the paste operation.
The paste operation should work approximately like this (the core concept in this proposal):
- At the start of a paste operation the states of DesktopKeyboard and GuestKeyboard are in sync, so the initial state of PasteKeyboard is a copy of the state of DesktopKeyboard.
- PasteKeyboard and GuestKeyboard are then together transitioned into the "idle" state by generating appropriate scancodes that release all keys pressed and unlock CapsLock in case it is locked. Having GuestKeyboard in idle state simplifies the scancode generation that follows next.
- The stream of scancodes is generated from the plaintext string and passed to the guest. Before a key scancode is sent it may be neccessary to send additional scancodes that toggle the state of the Shift and/or AltGr modifier keys. Since we track the keyboard state in PasteKeyboard we can reduce modifier key changes to a minimum (modifier keys are toggled lazily in PasteKeyboard).
- When a key on the system keyboard is pressed or released while a paste operation is in progress then only the state of DesktopKeyboard gets updated, but it is not otherwise reacted upon. An exception could be made for the Escape key to abort a long-running paste operation, or even that any key press aborts an ongoing paste operation.
- At the end of a paste operation it may be neccessary to generate scancodes to bring GuestKeyboard into sync with DesktopKeyboard (our main task). The states of DesktopKeyboard and GuestKeyboard may diverge in any way during a paste operation but must be in sync after it has finished.
Several problems and limitations arise when pasting strings into the guest:
- Generating scancodes from unicode characters depends on the visual keyboard layout configured in the guest's keyboard driver. The set of plaintext characters that can be pasted into the guest is limited to that of the selected keyboard layout.
- For each symbol used in the guest keyboard's visual layout we need a mapping from the symbol's unicode character to a sequence of pairs of (scancode, modifiers), where modifiers is a bitset of the Shift and the AltGr modifier keys (I believe Ctrl and Alt are reserved for OS and application use, but I'm not 100% sure). Normally only a single pair is needed, but dead keys cause sequences of multiple pairs. There are hundreds of different keyboard layouts worldwide, a reasonable subset for V86 needs to be selected and should be extended on demand. I have implemented these mappings in repository keyboard-tables, it's thoroughly tested (also see the interactive keyboard mapping explorer).
- The number of generated scancode bytes for toggling a single symbol can vary widely between 1 and 5 (maybe even more, see examples below).
- Generated scancode bytes cannot be sent to the guest at full speed, their transmission must be throttled in order to avoid losing scancodes due to a possible bottleneck in the guest's IRQ handler. I would suggest to send whole scancodes in bursts of maybe 8, 10 or 15 bytes and to pause somewhere between 50-100ms between bursts to give the guest time to digest the input. By "whole scancodes" I mean to not break up 16-bit scancodes across adjacent bursts. Burst size (in bytes) and delay (in milliseconds) should be configurable.
- A paste operation has a duration that depends on length and content of the string that is pasted, which may last for many seconds or even minutes (there's no real limit due to the unlimited string length). Only one paste operation may be active at a time.
Example scancode sequences for different keys and keyboard layouts:
"a" on a US keyboard generates 2 scancode bytes:
1E 9E
| |
| ~KeyA
KeyA
"Á" on a UK keyboard (Shift+AltGr+"A") generates 8 bytes:
2A E038 1E 9E AA E0B8
| | | | | |
| | | | | ~AltGr
| | | | ~Shift
| | | ~KeyA
| | KeyA
| AltGr
Shift
"Á" on a Swiss keyboard (dead key AltGr+"-" followed by Shift+"A") generates 10 bytes:
E038 0C 8C 2A E0B8 1E 9E AA
| | | | | | | |
| | | | | | | ~Shift
| | | | | | ~KeyA
| | | | | KeyA
| | | | ~AltGr
| | | Shift
| | ~Minus
| Minus
AltGr
Summary
This proposal is meant to define a robust and consistent paste function for V86 (I came up with this all by myself), let me know if it's too much :)
There is actually very little overlap between DesktopKeyboard and PasteKeyboard, they are backed by very different data structures and algorithms and only share how they keep track of their current keyboard state (the set of currently pressed keys).
Thanks for reading, any feedback is (as always) highly appreciated.
This is a draft proposal for a function that pastes strings into the guest by using the V86 keyboard with support for non-US keyboard layouts. This function is needed in different contexts:
This function is currently supported through
V86.keyboard_send_text(string, delay), but limited to the US keyboard layout.Desktop browser keyboard input
Let's start with a deep look at how V86 handles keyboard input in a desktop browser environment and with some needed terms and concepts. A simplified view of the keyboard pipeline when V86 is used with a desktop browser can be depicted as:
DesktopKeyboard represents the system keyboard state (the set of pressed keys), and GuestKeyboard the keyboard state as seen by the guest (GuestKeyboard is only a conceptual element which we don't implement). Our main task is to assert that the guest's keyboard state is always in sync with the system's keyboard state (except for key combinations that are intercepted by the OS or browser like Ctrl+Alt+Del).
A scancode is an 8- or 16-bit integer that represents the physical location of a key on a keyboard, a value that isn't altered by the visual keyboard layout or the state of the modifier keys. The 8th bit (0x80) of scancodes is reserved, when cleared it signals a "keydown" event, else a "keyup" event (these values are also called "make" and "break" codes, respectively).
Whenever the user presses or releases a key on the system keyboard its scancode and state is sent immediately to the V86 PS/2 controller and passed on to the guest's keyboard driver via interrupt. 16-bit scancodes are sent to the PS/2 controller as two separate bytes.
The desktop browser's
KeyboardEvent.codeproperty represents the same physical key location as a scancode but uses strings instead of integers to identify physical keys. "keydown" and "keyup" events are delivered in separateKeyboardEventinstances.A 1:1 mapping between KeyboardEvent.code and scancodes exists which gives a well-defined and universal solution for this, independent of the visual keyboard layout.
This is how V86 handles this case, though there is also a fallback to
KeyboardEvent.keyCodewhich only works for US-keyboards and has been deprecated in the standards 4 years ago or so, unless there are reasons to keep it I believe this fallback can be removed.String-based keyboard input
Even though the keyboard pipeline can be used to "paste" plaintext strings into the guest, this method works very differently than the one described above. The pipeline looks like this while a string paste operation is in progress:
DesktopKeyboard is muted here, meaning it still receives system keyboard events and updates its internal state but no scancodes are fed into the keyboard pipeline, the pipeline is instead fed by PasteKeyboard for the duration of the paste operation.
The paste operation should work approximately like this (the core concept in this proposal):
Several problems and limitations arise when pasting strings into the guest:
Example scancode sequences for different keys and keyboard layouts:
Summary
This proposal is meant to define a robust and consistent paste function for V86 (I came up with this all by myself), let me know if it's too much :)
There is actually very little overlap between DesktopKeyboard and PasteKeyboard, they are backed by very different data structures and algorithms and only share how they keep track of their current keyboard state (the set of currently pressed keys).
Thanks for reading, any feedback is (as always) highly appreciated.