Associate UHID devices to virtual displays on Android 14+ #6009

yume-chan · 2025-04-16T06:38:03Z

Only works on Android 14 and newer because it requires ASSOCIATE_INPUT_DEVICE_TO_DISPLAY permission.

When running multiple instances of scrcpy --display-id or scrcpy --new-display, each display will have its own mouse pointer.

Has no effect on keyboard, the whole system still only has one input focus.

I used FakeContext.get().getSystemService(Context.INPUT_SERVICE) to get InputManager, because initially I was testing addPortAssociation, which doesn't exist on InputManagerGlobal. This simplifies the code, especially for ClipboardManager, where we can use the stable public API (https://github.com/yume-chan/leap-scrcpy/blob/f9aaf1b05118261d75b82ba88b462f08e37eecdc/server/app/src/main/java/leap/scrcpy/server/FakeContext.kt#L62-L72)

rom1v · 2025-04-16T09:56:55Z

Thank you 👍 I will review later.

Just a quick question:

I used FakeContext.get().getSystemService(Context.INPUT_SERVICE) to get InputManager, because initially I was testing addPortAssociation, which doesn't exist on InputManagerGlobal.

Doesn't it cause a regression for an issue fixed by 5bd7514 on some Android versions?

yume-chan · 2025-04-16T12:15:15Z

Doesn't it cause a regression for an issue fixed by 5bd7514 on some Android versions?

I see #4074 is caused by ActivityThread.currentApplication() being null, but now there is a polyfilled one from

scrcpy/server/src/main/java/com/genymobile/scrcpy/Workarounds.java

Lines 119 to 134 in dc2fcc4

    
           private static void fillAppContext() { 
        
               try { 
        
                   Application app = new Application(); 
        
                   Field baseField = ContextWrapper.class.getDeclaredField("mBase"); 
        
                   baseField.setAccessible(true); 
        
                   baseField.set(app, FakeContext.get()); 
        
                   // activityThread.mInitialApplication = app; 
        
                   Field mInitialApplicationField = ACTIVITY_THREAD_CLASS.getDeclaredField("mInitialApplication"); 
        
                   mInitialApplicationField.setAccessible(true); 
        
                   mInitialApplicationField.set(ACTIVITY_THREAD, app); 
        
               } catch (Throwable throwable) { 
        
                   // this is a workaround, so failing is not an error 
        
                   Ln.d("Could not fill app context: " + throwable.getMessage()); 
        
               } 
        
           }

So If I change com.genymobile.scrcpy.wrappers.InputManager#create back to

    static InputManager create() {
        try {
            Object im = android.hardware.input.InputManager.class.getDeclaredMethod("getInstance").invoke(null);
            return new InputManager(im);
        } catch (NoSuchMethodException | IllegalAccessException | InvocationTargetException e) {
            throw new RuntimeException(e);
        }
    }

It still works (and calls FakeContext.get().getSystemService(Context.INPUT_SERVICE) through several layers of indirections)

rom1v · 2025-04-17T16:11:12Z

This simplifies the code, especially for ClipboardManager, where we can use the stable public API

I just tested to simplify ClipboardManager by using FakeContext.get().getSystemService(FakeContext.CLIPBOARD_SERVICE) as follow: 363c9a5

But then, when I paste:

java.lang.SecurityException: Package android does not belong to 2000
	at android.os.Parcel.createExceptionOrNull(Parcel.java:3261)
	at android.os.Parcel.createException(Parcel.java:3245)
	at android.os.Parcel.readException(Parcel.java:3228)
	at android.os.Parcel.readException(Parcel.java:3170)
	at android.content.IClipboard$Stub$Proxy.getPrimaryClip(IClipboard.java:459)
	at android.content.ClipboardManager.getPrimaryClip(ClipboardManager.java:253)
	at com.genymobile.scrcpy.wrappers.ClipboardManager.getText(ClipboardManager.java:27)
	at com.genymobile.scrcpy.device.Device.getClipboardText(Device.java:106)
	at com.genymobile.scrcpy.device.Device.setClipboardText(Device.java:119)
	at com.genymobile.scrcpy.control.Controller.setClipboard(Controller.java:568)
	at com.genymobile.scrcpy.control.Controller.handleEvent(Controller.java:279)
	at com.genymobile.scrcpy.control.Controller.control(Controller.java:192)
	at com.genymobile.scrcpy.control.Controller.lambda$start$1$com-genymobile-scrcpy-control-Controller(Controller.java:200)
	at com.genymobile.scrcpy.control.Controller$$ExternalSyntheticLambda1.run(D8$$SyntheticClass:0)
	at java.lang.Thread.run(Thread.java:1012)
Caused by: android.os.RemoteException: Remote stack trace:
	at android.app.AppOpsManager.checkPackage(AppOpsManager.java:9636)
	at com.android.server.clipboard.ClipboardService.clipboardAccessAllowed(ClipboardService.java:1329)
	at com.android.server.clipboard.ClipboardService.clipboardAccessAllowed(ClipboardService.java:1313)
	at com.android.server.clipboard.ClipboardService.-$$Nest$mclipboardAccessAllowed(ClipboardService.java:0)
	at com.android.server.clipboard.ClipboardService$ClipboardImpl.getPrimaryClip(ClipboardService.java:663)

yume-chan · 2025-04-17T16:15:23Z

For ClipboardManager, the mContext field needs to be replaced with FakeContext, see my link in OP: https://github.com/yume-chan/leap-scrcpy/blob/f9aaf1b05118261d75b82ba88b462f08e37eecdc/server/app/src/main/java/leap/scrcpy/server/FakeContext.kt#L62-L72

rom1v · 2025-04-17T16:24:31Z

Indeed, thank you, it works: a899752

rom1v · 2025-04-24T18:53:43Z

server/src/main/java/com/genymobile/scrcpy/control/UhidManager.java

@@ -52,13 +82,15 @@ public void open(int id, int vendorId, int productId, String name, byte[] report
        try {
            FileDescriptor fd = Os.open("/dev/uhid", OsConstants.O_RDWR, 0);
            try {
-                FileDescriptor old = fds.put(id, fd);
+                // Must be unique across the system
+                String inputPort = "scrcpy:" + Os.getpid() + ":" + id;


There is only one virtual display per scrcpy session, do we need one input port per UHID device?

I think we could just use String inputPort = "scrcpy:" + Os.getpid(); and call addUniqueIdAssociationByPort() only once (and all UHID devices would use the same input port associated to the single display id). That way, storing an inputPort per device would be unnecessary.

I'm not asking you to make any change, I would just like to know what you think about it.

(Currently, I cannot test virtual displays because of #5523 on my current device.)

I never thought about that. Initially I saw Android Studio is using different phys values for each device: https://cs.android.com/android-studio/platform/tools/adt/idea/+/mirror-goog-studio-main:streaming/screen-sharing-agent/app/src/main/cpp/virtual_input_device.cc;l=66-68;drc=fc06f05ca98ff7d3c8558976fd7eaf1b5480e2e8

But I believe multiple devices can have the same phys value, it means they are in one physical device, but with multiple input features. And addUniqueIdAssociationByPort still works.

The original commit has been backed up at https://github.com/yume-chan/scrcpy/tree/feat/associate-inpt-port-2

rom1v · 2025-04-25T09:21:22Z

(Currently, I cannot test virtual displays because of #5523 on my current device.)

I tested on two devices:

Motorola Edge 30 Neo with Android 14
Redmi Note 13 with Android 14

On both of them, running either your initial branch or your new one, the UHID mouse pointer does not appear on the virtual display (but it correctly disappears from the main display).

I run with:

scrcpy --new-display -M
scrcpy --new-display -M --start-app=org.videolan.vlc  # to get an app running

yume-chan · 2025-04-25T14:11:42Z

I see. So Android 14 still only has one mouse pointer across all displays. We need not only associate the UHID device to the display, but also move the mouse cursor to the display, which is impossible because the API is not available outside system server (same as my original comment #4829 (comment)).

I guess adding association alone is enough for virtual touchscreen devices

Only Android 15 added seperated mouse cursors for each display (https://cs.android.com/android/_/android/platform/frameworks/native/+/da10dd34cb41f0f117fe87f568f16f6025ab3d72)

rom1v · 2025-04-25T15:02:01Z

Thank you.

Here is a branch: pr6009.3 (not tested, because I currently have no device with Android 15 not impacted by #5523), with the following changes:

extract the change for using the public InputManager API to a separate commit
pass the display unique id in the "synchronized" DisplayData instance (DisplayManager can only be called from the "display" thread currently, as there is no synchronization, so pass the display unique id along with the display id and the position mapper)
wait for the virtual display to be available before creating the UhidManager (with a timeout of 1 second), using the same mechanism as for --start-app
handle the case where the unique id is null (in that case, do not associate an input port)
(as a side effect) also associate in case displayId == 0 (should we make a special case?)

TODO:

retrieve the display unique id also with the fallback mechanism in DisplayManager.parseDisplayInfo()

rom1v · 2025-04-26T08:56:57Z

TODO:

retrieve the display unique id also with the fallback mechanism in DisplayManager.parseDisplayInfo()

It would require to rework the parsing, as a single regex is not sufficient (depending on the device, the values are not always in the same order). Since it's just a fallback for some old devices, it is probably useless, so I will not do it until necessary.

So I think the branch can be considered ready.

Could you please review/test? I managed to test it successfully on a Pixel 6a.

yume-chan · 2025-04-29T12:38:39Z

The timing between adding port association and creating UHID devices doesn't matter, so I think it could be simplified a little (just adding the association when initializing UHID manager and remove it on exit).

The association will NOT be removed by Android OS when Scrcpy process exits, although it won't cause any harm, if we want to make sure it's 100% clean, maybe we need to add it to the CleanUp process.

Otherwise looks good to me and tests OK on my Redmi K70 Pro running Android 15.

rom1v · 2025-04-30T14:53:45Z

Thank you for your review.

The timing between adding port association and creating UHID devices doesn't matter, so I think it could be simplified a little (just adding the association when initializing UHID manager and remove it on exit).

When scrcpy runs without UHID device, I want to avoid adding a useless association. So it must be done only when at least one UHID device is created (and only once the virtual display id is known, whichever happens first).

Also, I think that my branch only adds the association for "new virtual displays" (not when --display-id=XX is passed).

What makes it more complicated is that currently, DisplayManager may not be called from the controller thread (and I don't want to add unnecessary synchronization, but maybe I'll have to so that I can get the displayUniqueId from a displayId in the controller thread, I will think about it).

rom1v · 2025-05-03T11:54:39Z

Also, I think that my branch only adds the association for "new virtual displays" (not when --display-id=XX is passed).

Fixed, now it should be called both for virtual displays and non-main display (I disabled the association for the main display, since it works by default).

What makes it more complicated is that currently, DisplayManager may not be called from the controller thread (and I don't want to add unnecessary synchronization, but maybe I'll have to so that I can get the displayUniqueId from a displayId in the controller thread, I will think about it).

I added synchronization and now call getDisplayInfo() from the controller thread.

The association will NOT be removed by Android OS when Scrcpy process exits, although it won't cause any harm, if we want to make sure it's 100% clean, maybe we need to add it to the CleanUp process.

OK. Since it is harmless, for simplicity I just remove the association when it closes normally, but not if the process is killed.

New version: pr6009.7

Use the public ClipboardManager API, with the FakeContext as context. PR #6009 <#6009> Suggested by: Simon Chan <1330321+yume-chan@users.noreply.github.com>

Use the public InputManager API. PR #6009 <#6009> Signed-off-by: Romain Vimont <rom@rom1v.com>

Do not use reflection to retrieve the method for every call. PR #6009 <#6009>

The DisplayManager and its method getDisplayInfo() may be used from both the Controller thread and the video (main) thread. PR #6009 <#6009>

This allows the mouse pointer to appear on the correct display (only for devices running Android 15+). Fixes #5547 <#5547> PR #6009 <#6009> Signed-off-by: Romain Vimont <rom@rom1v.com>

rom1v · 2025-05-22T18:04:40Z

I rebased: pr6009.8.

@yume-chan Could you please test and confirm that it works correctly before I merge? I was able to test with an older version of the branch, but the device I used has been updated and is now impacted by #5523).

yume-chan · 2025-06-01T08:47:18Z

I tested pr6009.8 works on my Redmi K70 Pro running Android 15.

rom1v · 2025-06-02T06:19:48Z

@yume-chan Thank you for your tests 👍

I just noticed that 3b36f29 breaks copy-paste from the Android device to the computer: the callback passed to clipboardManager.addPrimaryClipChangedListener() is never called.

anotheruserofgithub · 2025-06-03T15:20:42Z

@rom1v Just a side remark in Controller.waitDisplayData(long) called from Controller.getUhidManager():

scrcpy/server/src/main/java/com/genymobile/scrcpy/control/Controller.java

Lines 698 to 702 in e0f37f8

    
           long timeout = deadline - System.currentTimeMillis(); 
        
           if (timeout < 0) { 
        
               return null; 
        
           } 
        
           displayDataAvailable.wait(timeout);

It should test if (timeout <= 0) because if timeout is zero thenObject.wait(long) waits for a notification (or a spurious wakeup).

Other classes in the java.util.concurrent package behave differently when timeout is zero or negative, though you often have to delve into the implementation to be sure how they work. For instance, ReentrantLock.tryLock(long, TimeUnit) (source, called from here) will not synchronize and not wait at all, while if you create a Condition from a ReentrantLock then Condition.await(long, TimeUnit) (source, ConditionObject created here) will still yield the lock. It was more clearly documented e.g. in JDK 11 but comments disappeared in more recent versions.

rom1v reviewed Apr 24, 2025

View reviewed changes

Associate UHID devices to virtual displays on Android 14+

6d17ee6

yume-chan force-pushed the feat/associate-input-port branch from 0ef8f64 to 6d17ee6 Compare April 25, 2025 04:32

Only call addUniqueIdAssociationByPort on Android 15

7a76cb5

Benjamin-Loison mentioned this pull request May 4, 2025

Voice assistant Benjamin-Loison/android#28

Open

rom1v added a commit that referenced this pull request May 22, 2025

Simplify ClipboardManager wrapper

3b36f29

Use the public ClipboardManager API, with the FakeContext as context. PR #6009 <#6009> Suggested by: Simon Chan <1330321+yume-chan@users.noreply.github.com>

rom1v pushed a commit that referenced this pull request May 22, 2025

Simplify InputManager wrapper

384dcbb

Use the public InputManager API. PR #6009 <#6009> Signed-off-by: Romain Vimont <rom@rom1v.com>

rom1v added a commit that referenced this pull request May 22, 2025

Cache getDisplayInfo method

1b946ed

Do not use reflection to retrieve the method for every call. PR #6009 <#6009>

rom1v added a commit that referenced this pull request May 22, 2025

Synchronize access to DisplayManager

2f97112

The DisplayManager and its method getDisplayInfo() may be used from both the Controller thread and the video (main) thread. PR #6009 <#6009>

Uh oh!

Associate UHID devices to virtual displays on Android 14+ #6009

Are you sure you want to change the base?

Associate UHID devices to virtual displays on Android 14+ #6009

Conversation

yume-chan commented Apr 16, 2025

Uh oh!

rom1v commented Apr 16, 2025

Uh oh!

yume-chan commented Apr 16, 2025

Uh oh!

rom1v commented Apr 17, 2025

Uh oh!

yume-chan commented Apr 17, 2025

Uh oh!

rom1v commented Apr 17, 2025

Uh oh!

rom1v Apr 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yume-chan Apr 25, 2025

Choose a reason for hiding this comment

Uh oh!

rom1v commented Apr 25, 2025

Uh oh!

yume-chan commented Apr 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rom1v commented Apr 25, 2025

Uh oh!

rom1v commented Apr 26, 2025

Uh oh!

yume-chan commented Apr 29, 2025

Uh oh!

rom1v commented Apr 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rom1v commented May 3, 2025

Uh oh!

rom1v commented May 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

yume-chan commented Jun 1, 2025

Uh oh!

rom1v commented Jun 2, 2025

Uh oh!

anotheruserofgithub commented Jun 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

rom1v Apr 24, 2025 •

edited

Loading

yume-chan commented Apr 25, 2025 •

edited

Loading

rom1v commented Apr 30, 2025 •

edited

Loading

rom1v commented May 22, 2025 •

edited

Loading

anotheruserofgithub commented Jun 3, 2025 •

edited

Loading