New IO keyboard/mouse/gamepad event API (1.87) recap

[ocornut]

EDIT 2022/11/15: Dear ImGui 1.89 made ImGuiKey a strongly-typed enum, which broke backward compatibility with using legacy backend-specific indices (e.g. IsKeyPressed(GLFW_KEY_A) needs casting IsKeyPressing((ImGuiKey)GLFW_KEY_A). I am not entirely sure this was the right decision as this is unusually breaking, but fix is trivial and should encourage moving to new API so leaving the strongly-typed enum for now.

---

EDIT 2022/02/29: We discovered a backward-compatibility regression in 18700:

• Direct legacy reads from io.KeysDown[] won't work with new backends.
• And in particular io.KeysDown[GetKeyIndex[XXX]) will buffer overflow in old and new backends.
  Reads via IsKeyDown() function are always working, this only affect direct array access This was fixed in 18708 (commit 5659db5)

---

(This is a followup on #4858 now that things have settled)

We (@thedmd @ocornut) have completely revamped IO API in 1.87 (the way backend communicate with dear imgui to submit inputs)

All backends have been updated to use new API.

Both user code and old backends should be backward compatible (apart from a few rare edge cases) but we encourage you to transition.

Additions:

• New functions: io.AddKeyEvent(), io.AddKeyAnalogEvent(), io.AddKeyModsEvent() removed jan 27, io.AddMousePosEvent(), io.AddMouseButtonEvent(), io.AddMouseWheelEvent(), io.SetKeyEventNativeData() (for legacy support).
• New enums: ImGuiKey now contains a full keyset (e.g. ImGuiKey_F1) this allow using full range of keyboard inputs in a portable manner, which was previously impossible as we encouraged use of native indices. This will make it easier for people to share code/extensions publicly or across multiple codebases.
• New enums: ImGuiKey also contains gamepad axises/buttons ImGuiKey_GamepadDpadUp) unifying input sources.
• Input queue now trickle conflicting inputs over multiple frames (unless io.ConfigInputTrickleEventQueue == false) which massively improve input experience with low framerate.
• Both SDL and GLFW backends have been fixed to submit translated keys, so ImGuiKey_A will be submitted when pressing the key which user would need to press to emit an "A" character, allowing for translated shortcuts.

Those changes will constitute the bulk of 1.87.
If you want to help, please upgrade today :)

Transition Guide

If you are using a standard backend (RECOMMENDED) just update it to latest!
If you have are using a custom backend:

Keyboard:

• Backend writing to io.KeyMap[], io.KeysDown[] -> backend should call io.AddKeyEvent()
• If your backend now call io.AddKeyEvent() and you want legacy user code to still work with legacy indices, also call io.SetKeyEventNativeData().
• User code calling IsKeyPressed(MY_NATIVE_KEY_XXX) -> call IsKeyPressed(ImGuiKey_XXX) (OLD CODE WILL STILL WORK*)
• User code calling IsKeyPressed(GetKeyIndex(ImGuiKey_XXX)) -> call IsKeyPressed(ImGuiKey_XXX) (OLD CODE WILL STILL WORK*)
• All keyboard related functions taking int user_key_index now take ImGuiKey key: IsKeyDown(), IsKeyPressed(), IsKeyReleased(), GetKeyPressedAmount().
• Basically the trick we took advantage of is that we previously only supported native keycode from 0 to 511, so ImGuiKey values can still express a legacy native keycode, and new named keys are all >= 512. Luckily that old constraint allowed us to save the day!
• Added GetKeyName() helper function.
• Obsoleted GetKeyIndex(): it is now unnecessary and will now return the same value.
• For all new calls to IO functions, the Dear ImGui context should be bound/current.

(* until IMGUI_DISABLE_OBSOLETE_KEYIO is set. In a few versions, IMGUI_DISABLE_OBSOLETE_FUNCTIONS will automatically set it)
(since 1.90, defining IMGUI_DISABLE_OBSOLETE_FUNCTIONS automatically defines IMGUI_DISABLE_OBSOLETE_KEYIO.)
(* and unless the backend forgot to call io.SetKeyEventNativeData())

Mouse:

• Backend writing to io.MousePos -> backend should call io.AddMousePosEvent()
• Backend writing to io.MouseDown[] -> backend should call io.AddMouseButtonEvent()
• Backend writing to io.MouseWheel, io.MouseWheelH -> backend should call io.AddMouseWheelEvent()
• (Docking Branch w/ Multi-viewports) Backend writing to io.MouseHoveredViewport-> backend should call io.AddMouseViewportEvent()
• For all new calls to IO functions, the Dear ImGui context should be bound/current.

Gamepad/Nav:

• Backend writing to io.NavInputs[] -> backend should call io.AddKeyEvent()/io.AddKeyAnalogEvent() with ImGuiKey_GamepadXXX values.

What is input queue trickling?

TL;DR; if you submit "Key A went down, then went up, then went down, then went up" during the same Dear ImGui frame, previously events would be missed and it was the application/backend responsibility to submit those events wisely. We moved that responsibility to the core library now. Because most of Dear ImGui code and widgets are reading inputs levels (rather than events), we spread those events over multiple frames when necessary. This makes dear imgui easier to use at low framerate (e.g. 10 FPS).

Backends polling inputs vs handling events

One gotcha is that if you submit events it is preferable to submit then in the right order.
If you transition to new API make sure you are submitting events in the order they come from your OS / platform layer. Normally this should be the default. But because we are punks, our old backends had a tendency to be polling some input state instead of reacting to events, which effectively "lost" their order (e.g. which have changed first, Mouse Position or Mouse Button state?). We have now transitioned all standard backends to be reacting on events rather than polling. If you used your own backend maybe you already behaved better.

Demo

You can see pressed keys in the demo + drawing a small section of a keyboard for no other reason that it being fun.

Reference changes (e.g. to platform backends)

ALWAYS REFER TO FINAL VERSION AS SOME MINOR STUFF MAY HAVE BEEN RENAMED/CHANGED SINCE THE INTERMEDIARY COMMITS. Those references commits are here to help understanding what changed, if you need to update a custom backend.

Core lib

• Core: Rename ImGuiKey_KeyPadEnter > ImGuiKey_KeypadEnter (afffcd5)
• Core: Add extra keys, io.AddKeyEvent() and more (3b66929, bf08c13)
• Core: Add io.AddKeyModsEvent() and updated all Backends accordingly. (790132a, e8172fd)
• Core: Add AddMousePosEvent(), AddMouseButtonEvent(), AddMouseWheelEvent() api + updated all Backends. (b8e56dc)
• Core: Add Input Queue with automatic trickling of fast events. (7374b96, 90a6961, 7ad42ff)
• Core: Add AddKeyAnalogEvent() and support for ImGuiKey_GamepadXXXX. (f33bb99)

Win32 backend

• Win32: update to use io.AddKeyEvent(), add full key map (746c9f7)
• Win32: maintain MouseButtonsDown mask instead of using IsAnyMouseDown() which will be obsoleted (7f8a89c)
• Win32: reorganize to update mouse inputs using WM_MOUSEMOVE/WM_MOUSELEAVE instead of polling so we can submit events in the right order + fallback to provide it when focused but not hovered/captured + update MousePos before Key Modifiers (bf4de2a)
• Win32: update to use AddMousePosEvent(), AddMouseButtonEvent(), AddMouseWheelEvent(). (b8e56dc)
• Win32: add full gamepad support using io.AddKeyEvent(), io.AddKeyAnalogEvent(), stopped writing to io.NavInputs[]. (9f8c599)

GLFW backend

• GLFW: submitting translated keys (100ede5)
• GLFW: update to use io.AddKeyEvent(), add full key map (ecd212c)
• GLFW: reorganize to update mouse inputs using glfwSetCursorPosCallback() (breaking if user install their callback themselves) instead of polling so we can submit events in the right order + fallback to provide it when focused but not hovered/captured + update MousePos before MouseButtons. (200a8f1)
• GLFW: update to use AddMousePosEvent(), AddMouseButtonEvent(), AddMouseWheelEvent(). (b8e56dc)
• GLFW: add full gamepad support using io.AddKeyEvent(), io.AddKeyAnalogEvent(), stopped writing to io.NavInputs[]. (3d85433)

SDL backend

• SDL: update to use io.AddKeyEvent(), add full key map (fe646ea)
• SDL: maintain MouseButtonsDown mask instead of using IsAnyMouseDown() which will be obsoleted (7f8a89c)
• SDL: reorganize to update mouse inputs using SDL_MOUSEMOTION/SDL_WINDOWEVENT_LEAVE instead of polling so we can submit events in the right order + fallback to provide it when focused but not hovered/captured + update MousePos before MouseButtons (98ce013).
• SDL: update to use AddMousePosEvent(), AddMouseButtonEvent(), AddMouseWheelEvent(). (b8e56dc)
• SDL: add full gamepad support using io.AddKeyEvent(), io.AddKeyAnalogEvent(), stopped writing to io.NavInputs[]. (39c3412)

Other backends

• OSX: update to use io.AddKeyEvent(), add full key map (ee436aa)
• OSX: add full gamepad support using io.AddKeyEvent(), io.AddKeyAnalogEvent(), stopped writing to io.NavInputs[]. (5ea47d9)
• Android: update to use io.AddKeyEvent(), add full key map (1797135)
• Allegro: update to use io.AddKeyEvent(), add full key map (1bfe4a7)
• GLUT: update to use io.AddKeyEvent(), add full key map (da1864d)

List of keys

ImGuiKey_None = 0,
ImGuiKey_Tab = 512,             // == ImGuiKey_NamedKey_BEGIN
ImGuiKey_LeftArrow,
ImGuiKey_RightArrow,
ImGuiKey_UpArrow,
ImGuiKey_DownArrow,
ImGuiKey_PageUp,
ImGuiKey_PageDown,
ImGuiKey_Home,
ImGuiKey_End,
ImGuiKey_Insert,
ImGuiKey_Delete,
ImGuiKey_Backspace,
ImGuiKey_Space,
ImGuiKey_Enter,
ImGuiKey_Escape,
ImGuiKey_LeftCtrl,
ImGuiKey_LeftShift,
ImGuiKey_LeftAlt,
ImGuiKey_LeftSuper,
ImGuiKey_RightCtrl,
ImGuiKey_RightShift,
ImGuiKey_RightAlt,
ImGuiKey_RightSuper,
ImGuiKey_Menu,
ImGuiKey_0,
ImGuiKey_1,
ImGuiKey_2,
ImGuiKey_3,
ImGuiKey_4,
ImGuiKey_5,
ImGuiKey_6,
ImGuiKey_7,
ImGuiKey_8,
ImGuiKey_9,
ImGuiKey_A,
ImGuiKey_B,
ImGuiKey_C,
ImGuiKey_D,
ImGuiKey_E,
ImGuiKey_F,
ImGuiKey_G,
ImGuiKey_H,
ImGuiKey_I,
ImGuiKey_J,
ImGuiKey_K,
ImGuiKey_L,
ImGuiKey_M,
ImGuiKey_N,
ImGuiKey_O,
ImGuiKey_P,
ImGuiKey_Q,
ImGuiKey_R,
ImGuiKey_S,
ImGuiKey_T,
ImGuiKey_U,
ImGuiKey_V,
ImGuiKey_W,
ImGuiKey_X,
ImGuiKey_Y,
ImGuiKey_Z,
ImGuiKey_F1,
ImGuiKey_F2,
ImGuiKey_F3,
ImGuiKey_F4,
ImGuiKey_F5,
ImGuiKey_F6,
ImGuiKey_F7,
ImGuiKey_F8,
ImGuiKey_F9,
ImGuiKey_F10,
ImGuiKey_F11,
ImGuiKey_F12,
ImGuiKey_F13, 
ImGuiKey_F14, 
ImGuiKey_F15, 
ImGuiKey_F16, 
ImGuiKey_F17, 
ImGuiKey_F18,
ImGuiKey_F19, 
ImGuiKey_F20, 
ImGuiKey_F21, 
ImGuiKey_F22, 
ImGuiKey_F23, 
ImGuiKey_F24,
ImGuiKey_Apostrophe,        // '
ImGuiKey_Comma,             // ,
ImGuiKey_Minus,             // -
ImGuiKey_Period,            // .
ImGuiKey_Slash,             // /
ImGuiKey_Semicolon,         // ;
ImGuiKey_Equal,             // =
ImGuiKey_LeftBracket,       // [
ImGuiKey_Backslash,         // \ (this text inhibit multiline comment caused by backslash)
ImGuiKey_RightBracket,      // ]
ImGuiKey_GraveAccent,       // `
ImGuiKey_CapsLock,
ImGuiKey_ScrollLock,
ImGuiKey_NumLock,
ImGuiKey_PrintScreen,
ImGuiKey_Pause,
ImGuiKey_Keypad0,
ImGuiKey_Keypad1,
ImGuiKey_Keypad2,
ImGuiKey_Keypad3,
ImGuiKey_Keypad4,
ImGuiKey_Keypad5,
ImGuiKey_Keypad6,
ImGuiKey_Keypad7,
ImGuiKey_Keypad8,
ImGuiKey_Keypad9,
ImGuiKey_KeypadDecimal,
ImGuiKey_KeypadDivide,
ImGuiKey_KeypadMultiply,
ImGuiKey_KeypadSubtract,
ImGuiKey_KeypadAdd,
ImGuiKey_KeypadEnter,
ImGuiKey_KeypadEqual,
ImGuiKey_AppBack,               // Available on some keyboard/mouses. Often referred as "Browser Back"
ImGuiKey_AppForward,

// Gamepad (some of those are expected to be analog values from 0.0f to 1.0f) ..............// NAVIGATION action
ImGuiKey_GamepadStart,          // Menu (Xbox)          + (Switch)      Start/Options (PS)  // --
ImGuiKey_GamepadBack,           // View (Xbox)          - (Switch)      Share (PS)          // --
ImGuiKey_GamepadFaceUp,         // Y (Xbox)             X (Switch)      Triangle (PS)       // -> ImGuiNavInput_Input
ImGuiKey_GamepadFaceDown,       // A (Xbox)             B (Switch)      Cross (PS)          // -> ImGuiNavInput_Activate
ImGuiKey_GamepadFaceLeft,       // X (Xbox)             Y (Switch)      Square (PS)         // -> ImGuiNavInput_Menu
ImGuiKey_GamepadFaceRight,      // B (Xbox)             A (Switch)      Circle (PS)         // -> ImGuiNavInput_Cancel
ImGuiKey_GamepadDpadUp,         // D-pad Up                                                 // -> ImGuiNavInput_DpadUp
ImGuiKey_GamepadDpadDown,       // D-pad Down                                               // -> ImGuiNavInput_DpadDown
ImGuiKey_GamepadDpadLeft,       // D-pad Left                                               // -> ImGuiNavInput_DpadLeft
ImGuiKey_GamepadDpadRight,      // D-pad Right                                              // -> ImGuiNavInput_DpadRight
ImGuiKey_GamepadL1,             // L Bumper (Xbox)      L (Switch)      L1 (PS)             // -> ImGuiNavInput_FocusPrev + ImGuiNavInput_TweakSlow
ImGuiKey_GamepadR1,             // R Bumper (Xbox)      R (Switch)      R1 (PS)             // -> ImGuiNavInput_FocusNext + ImGuiNavInput_TweakFast
ImGuiKey_GamepadL2,             // L Trigger (Xbox)     ZL (Switch)     L2 (PS) [Analog]
ImGuiKey_GamepadR2,             // R Trigger (Xbox)     ZR (Switch)     R2 (PS) [Analog]
ImGuiKey_GamepadL3,             // L Thumbstick (Xbox)  L3 (Switch)     L3 (PS)
ImGuiKey_GamepadR3,             // R Thumbstick (Xbox)  R3 (Switch)     R3 (PS)
ImGuiKey_GamepadLStickUp,       // [Analog]                                                 // -> ImGuiNavInput_LStickUp
ImGuiKey_GamepadLStickDown,     // [Analog]                                                 // -> ImGuiNavInput_LStickDown
ImGuiKey_GamepadLStickLeft,     // [Analog]                                                 // -> ImGuiNavInput_LStickLeft
ImGuiKey_GamepadLStickRight,    // [Analog]                                                 // -> ImGuiNavInput_LStickRight
ImGuiKey_GamepadRStickUp,       // [Analog]
ImGuiKey_GamepadRStickDown,     // [Analog]
ImGuiKey_GamepadRStickLeft,     // [Analog]
ImGuiKey_GamepadRStickRight,    // [Analog]

// Keyboard Modifiers
// - This is mirroring the data also written to io.KeyCtrl, io.KeyShift, io.KeyAlt, io.KeySuper, in a format allowing
//   them to be accessed via standard key API, allowing calls such as IsKeyPressed(), IsKeyReleased(), querying duration etc.
// - Code polling every keys (e.g. an interface to detect a key press for input mapping) might want to ignore those
//   and prefer using the real keys (e.g. ImGuiKey_LeftCtrl, ImGuiKey_RightCtrl instead of ImGuiKey_ModCtrl).
// - In theory the value of keyboard modifiers should be roughly equivalent to a logical or of the equivalent left/right keys.
//   In practice: it's complicated; mods are often provided from different sources. Keyboard layout, IME, sticky keys and
//   backends tend to interfere and break that equivalence. The safer decision is to relay that ambiguity down to the end-user...
ImGuiKey_ModCtrl,
ImGuiKey_ModShift,
ImGuiKey_ModAlt,
ImGuiKey_ModSuper,

ImGuiKey_COUNT,                 // No valid ImGuiKey is ever greater than this value

Questions?

(Will be updated over time)

Q: It is ok to call io.AddXXXEvent() functions every frame with unchanged data?
A: Yes.

[RT2Code]

Awesome changes! I'm in the process of updating my custom backend to this API, it will integrate much better in my callback based event system.

One thing though, If I understand correctly, ImGuiKey_Menu is supposed to represent this key :

If yes, then the SDL backend should use SDLK_APPLICATION instead of SDLK_MENU.

[thedmd]

Yes, this is a bug. It should be SDLK_APPLICATION.

[Flix01]

[OT] Not sure if somebody's interested... but when I saw the new "keyboard rectangle" in imgui_demo.cpp, I've soon tried to "extend" it. Here is what I've done so far [gist link].

show/hide screenshot

[thedmd]

I love it. I'm also so picky creature that I differentiate between ANSI and ISO layouts. :)

[ocornut]

IMPORTANT: If you have updated backends between January 10 and January 27
We have made a breaking change to the new IO api c906c65
io.AddKeyModsEvent() was replaced with normal calls to io.AddKeyEvent() with ImGuiKey_ModXXX values.

All backends have been updated. This only applies to you if you updated standard or custom backend between January 10 and January 27. If you use standard backends, simply update. If you use a custom backend change code above.