Feat/add remote desktop direct mode on wayland

[petit-aigle]

Before Submitting This PR

Please confirm you have done the following:

• I have searched existing issues and pull requests (including closed ones) to ensure this isn't a duplicate
• I have read CONTRIBUTING.md

If this is a feature or change that was previously closed/rejected:

• I have explained in the description below why this should be reconsidered
• I have gathered community feedback (link to discussion below)

---

Human Written Description

This PR introduces a Wayland-native keyboard injection path implemented on top of the GNOME Remote Desktop portal.
It adds a fully portal-backed flow for direct typing, including explicit user authorization, permission lifecycle management, and compositor-mediated input injection.

The objective is to enable direct text injection on Wayland using the mechanisms explicitly designed for it, while keeping the interaction constrained to a well-defined, user-approved security boundary.

Today, common Linux workarounds rely on installing tools such as ydotool, adding the user to the input group, and deploying permissive udev rules that grant write access to /dev/uinput. This setup enables global input injection for any process owned by the user, permanently widening the local attack surface and bypassing Wayland’s isolation guarantees.

By contrast, the portal-based approach keeps input injection scoped, revocable, and mediated by the compositor itself. The trust boundary remains explicit and auditable, aligning with Wayland’s original security goals rather than working against them (see discussion: #522 (comment)).

---

Related Issues / Discussions

Fixes #
Discussion: #522 (comment)

---

Community Feedback

No dedicated feedback thread yet.
This change is intentionally scoped to a specific security and architectural axis; broader feedback can be gathered after initial review if needed.

---

⚠️ Must be fixed (before merge)

• Some characters with symbols still end up in the wrong position in the string
• Some characters with symbols don’t appear at all (e.g. “ô”, “û”)
• Potential issue with uppercase letters or numbers (not reproducible on my side yet, waiting for confirmation from the reporter (@delthas) to see whether the issue still persists with the new version)

Information about previous fixes

The initial problem was that some accented characters were arriving “split”: the letter on one side and the accent on the other (e.g., “é” sent as U+0065 for “e” then U+0301 for the acute accent). Sent separately, the accent was mispositioned or lost. I therefore glued these pieces back together before sending, so each accented character is typed in a single shot.

That introduced another issue: if the active keyboard language doesn’t include a character (e.g., a layout without accents), pressing that character is ignored.

To work around this, I now use the Ctrl+Shift+U method followed by the character’s Unicode code, then validation. This types the symbol directly and works regardless of the chosen keyboard language. I think this is the best solution currently available for managing different keyboard configurations.

---

Testing

Manual tests only

Authorization, Token & UI Consistency

• Permission request accepted via the portal and correctly reflected in the UI
• Permission request refused via the portal and correctly reflected in the UI
• Token revocation while the application is running, with immediate UI state update
• Token revocation while the application is closed, followed by application restart and verification that:
  • the stored token is checked against the Flatpak permission store at startup
  • the application state is updated accordingly
  • the UI initializes consistently with the result of this check
• Consistent synchronization between in-memory state, persisted settings, and Flatpak portal permissions
• Verification of correct character injection when the portal authorization is present

Useful commands (Flatpak permission store – Remote Desktop)

⚠️ Warning: the commands below operate on the Flatpak permission store for the Remote Desktop portal.
Some of them can remove all Remote Desktop authorizations, not just the ones created by Handy.
Always inspect the current state before running destructive commands, and only proceed if you are sure no other Remote Desktop permissions are in use.

• flatpak permissions remote-desktop
  Lists all entries stored in the Flatpak permission table for the Remote Desktop portal.

• flatpak permission-remove remote-desktop <OBJECT_ID>
  Removes the Remote Desktop portal permission for a single specific object/application ID.
  This is the recommended and safe way to revoke one authorization without affecting others.

• flatpak permissions remote-desktop | awk '{print $2, ($3=="-"?"":$3)}' | xargs -n2 sh -c 'flatpak permission-remove remote-desktop "$0"'
  Iterates over the Flatpak permission table and removes multiple entries via flatpak permission-remove.
  ❗Highly destructive ❗: this can wipe all Remote Desktop permissions at once.

Flatpak portal permissions rely on a stable application identifier derived from desktop integration (i.e. a .desktop entry).
In an installed build, this identifier exists and allows Flatpak Permissions to associate Remote Desktop authorizations with the application.
In a development context, this desktop integration does not exist, so no stable application ID can be resolved and the permission entry may be empty.
This is expected behavior of the Flatpak permission system in dev environments, not a bug in the implementation.

---

Platform Coverage

• Linux (GNOME Wayland)
• Linux (non-Wayland / X11)
• macOS
• Windows

---

Screenshots / Videos

Screenshots showing:

• Permission request UI and live authorization state
  

Same source, video format -> handy_wayland_remote_desktop_permission_ui_video
previous version, before the UI update -> handy_wayland_remote_desktop_permission_ui_video

• Active typing session indicator (GNOME screen-sharing icon during injection)

Note: The GNOME screen-sharing indicator appears during injection and may linger briefly after completion. This is expected portal behavior and may warrant a short user-facing note in settings.

---

AI Assistance

• AI was used

If AI was used:

• Tools used: ChatGPT, Codex
• How extensively: AI tools were used as assistance to get start
  ed and to accelerate development. Codex was used in particular to bootstrap and navigate Rust-specific aspects, as I am an experienced Python developer but not a Rust specialist. AI was also used to help generate and translate the newly introduced i18n strings across all supported languages. Each portion of source code generated by AI was analyzed, validated, maîtrisée, challenged, and modified where necessary to ensure a high level of code quality. All architectural and technical decisions were made by me, and I remain fully responsible for the final design, implementation choices, and code.

[cjpais]

I think we should try to unify the UI for permissions giving into the onboarding similar to how it's done on Mac. Otherwise I'm excited about this PR

[petit-aigle]

@cjpais Thanks for the feedback, glad to hear you’re excited about the PR 🙂

I’m not entirely sure what you’re referring to regarding the macOS onboarding flow. I don’t currently have access to a Mac, so I haven’t been able to observe or test how permissions are presented there in this project.

Otherwise, since opening the merge request, I’ve fixed two issues: first, a wording/translation issue where “GNOME Wayland” was used instead of simply “Wayland”, even though Wayland can be used outside of GNOME. Second, I fixed the handling of accented characters, which were previously not supported and are now correctly handled.

Despite this fix, two issues remain:

1. some accents still end up in the wrong position in the string
2. some accented characters don’t appear at all, e.g., “ô” and “û”

I don’t yet have a reliable fix that doesn’t turn into a mess of edge-case handling.

[cjpais]

@petit-aigle you can look at the UI code for this, even if you are unfamiliar. It should be relatively trivial to have it appear also for linux.

[bkanuka]

FYI I pulled this down to test (couldn't get anything else to work in Ubuntu + Gnome + Wayland) and this worked perfectly! I also agree that this is the "right way" of doing things. Great work @petit-aigle 🚀

[cjpais]

@bkanuka thanks for the confirmation, I'll probably be taking a look today and trying to merge

[delthas]

The README should probably also be updated? Namely https://github.com/cjpais/Handy?tab=readme-ov-file#major-issues-help-wanted and https://github.com/cjpais/Handy?tab=readme-ov-file#linux-notes

[delthas]

Tested on my end, works fine on GNOME, except that many/most characters are always dropped (running with a French keyboard layout and French words): all upper case A to Z, some diacritic letters â, ê (but é and è work), Ç (but ç works), Ê, Â, also numbers 0123456789, and some puncutation . (but , and ' work).

(Initially I thought the first letter of all my inputs was dropped, but that was just because the first letter was always capital, and upper case letters are dropped on my system.)

Possibly more but that's what I was able to find.

[bkanuka]

Interesting! I didn't have that experience at all. Capitals, numbers, punctuation, all came through - but I only speak English

@delthas could you test in English? Setting gnome to a English layout? It would be interesting to know if that's the reason things are dropped for you.

[petit-aigle]

@delthas @bkanuka
@cjpais Just to reiterate, please don’t merge yet as mentioned earlier. These issues are still present. I’m going to look into this as soon as I have a bit of time.

Despite this fix, two issues remain:

1. some accents still end up in the wrong position in the string
2. some accented characters don’t appear at all, e.g., “ô” and “û”

I don’t yet have a reliable fix that doesn’t turn into a mess of edge-case handling.

[petit-aigle]

Okay, I've fixed the problem normally.

The initial problem was that some accented characters were arriving “split”: the letter on one side and the accent on the other (e.g., “é” sent as U+0065 for “e” then U+0301 for the acute accent). Sent separately, the accent was mispositioned or lost. I therefore glued these pieces back together before sending, so each accented character is typed in a single shot.

That introduced another issue: if the active keyboard language doesn’t include a character (e.g., a layout without accents), pressing that character is ignored.

To work around this, I now use the Ctrl+Shift+U method (example: Ctrl+Shift+U, then 00E9, then Enter → é) followed by the character’s Unicode code, then validation. This types the symbol directly and works regardless of the chosen keyboard language. I think this is the best solution currently available for managing different keyboard configurations.

@delthas I wasn’t able to reproduce the issue with numbers or with uppercase letters. I am also on GNOME with a French keyboard. Could you test the latest solution and let me know whether the other issues have already been fixed on your side, and whether the problems with numbers and uppercase letters still persist ? Thank you.

[delthas]

Hi, I pulled, reran bun run tauri dev and am still getting the exact same issues with some characters not working (all upper case A to Z, some diacritic letters â, ê (but é and è work), Ç (but ç works), Ê, Â, also numbers 0123456789, and some puncutation . (but , and ' work)).

[delthas]

FWIW I'm running the "French" layout (not "alt.", "legacy", etc.)

[petit-aigle]

@delthas
Ok, based on my investigation and your explanation, I have the impression that the keysyms (sent letters) only work if they actually represent a real physical combination on your keyboard.

Example:
Shift + a = A

Whereas normally, the portal (what is used to transmit key presses to Wayland) is supposed to accept the final character directly, without requiring a physical combination.
This would explain your behavior for everything related to uppercase letters and numbers.

But after the update, if that were the case, then even accented characters should no longer work, because special characters are sent as non-ASCII, so all accented characters are sent via their Unicode representation.

So for example using Ctrl + Shift + U followed by the Unicode value and then Enter.
Based on your examples, this gives

• Ctrl + Shift + U and ea and Enter for ê
• Ctrl + Shift + U and e8 and Enter for è
• Ctrl + Shift + U and e9 and Enter for é
• Ctrl + Shift + U and c7 and Enter for Ç
• Ctrl + Shift + U and c2 and Enter for Â
• Ctrl + Shift + U and cA and Enter for Ê

Basically, what this says is that è and é work, so sending 8 and 9 worked, but they would not work when they are requested directly.

Then why do the others not work?

At this point I’m a bit lost. As I told you, I have the exact same setup as you, so I’m missing some information to go further.

Could you please tell me which portal you are using?
You can run:

ps -ef | grep xdg-desktop-portal-

to see the list of installed portals.

Also, in:

~/.config/xdg-desktop-portal/portals.conf

check whether you have any specific configuration.

With:

/usr/libexec/xdg-desktop-portal --verbose

you should normally see which portal is actually loaded.

Could you also please double-check that you have pulled the latest version, for example by verifying that:

git log -1

and that the latest commit is indeed:

4a00b6e2bc473234d3ae673eb2d833a09f4ca0df

Sorry to ask all this. I know there is almost no chance you are not on the correct version, but the fact that é and è work really makes it look like it might be the case.

Also, when you launch the new version, please make sure to kill any Handy instance that might still be running in the background.

[petit-aigle]

@cjpais Hey !
I’ve updated the UI to take the “Typing Tool” into account (i’ve updated the GIF on the PR). I’m waiting for feedback from @delthas, but otherwise everything looks good on my side. I also took the opportunity to clean up the code a bit and better separate the different concerns.

[cjpais]

I'm not super happy with the UI flow. Mostly because permissions are generally handled during onboarding (as mentioned before). I understand why they are here, but just wondering if you took a look or thought through how it could be done during the onboarding.

Also be careful with colors, the app pretty much uses exclusively Handy colors and best to consider this. The UI is generally a bit rough and not matching the rest of the app.

[petit-aigle]

Ah yeah, that’s true, I’ll look into it as soon as possible, with all the small issues that came up in the meantime, I forgot to get back to this.

[cjpais]

No worries thanks

[panpanpanpaan]

Please don't do it this way, there is an xdg global shortcuts portal for this kind of stuff: https://flatpak.github.io/xdg-desktop-portal/docs/doc-org.freedesktop.portal.GlobalShortcuts.html

[bkanuka]

@panpanpanpaan The GlobalShorcuts portal is for registering keyboard shortcuts though (right?). The Remote Desktop portal is an acceptable way of allowing a program to act on the user's behalf (in this case, "typing on a keyboard").