-
-
Notifications
You must be signed in to change notification settings - Fork 10.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[WIP] UHID mouse/keyboard input #4473
Conversation
Thank you very much 👍 Sorry for the delay, I don't have much time currently. After a quick glance, it looks good 👍, but I need to review in more details to better understand how everything works. One problem I encounter however is that it does not use the correct keyboard mapping. If I use AOA, I can select the physical keyboard mapping once (doc | #2632 (comment)), and it works fine. But with UHID it is not used, so when I press |
Keyboard layout setting is also available for UHID virtual keyboards. I think Android system can distinguish different keyboards and save different layout settings for each device. The display name is set here: https://github.com/Genymobile/scrcpy/pull/4473/files#diff-ee5bd2180bba9b212ca2eb2799d4272e64029b26751a4e305beac776124aee93R25 |
Awesome 👌 |
I added keyboard output report parsing. After the virtual keyboard is registered, or when keyboard modifier status changes, the device will send an output report for keyboard LED status. The client can combine this information with computer keyboard modifier key status to make sure they are in sync. For example, if the user toggles caps lock outside Scrcpy, on the next key event, the client can detect the local modifier status is different from what is on device, and send an extra caps lock key press event to correct that. Or, if caps lock is on on device but off on PC when Scrcpy starts, the client can turn off caps lock on device. This might resolve #4350. |
As a preliminary work, I started to extract the replacement of But I think that Maybe |
I think only |
One concrete example could be an option to install and switch to a specific keyboard on the device (AdbKeyboard, NullKeyboard…). Even if such an option is not planned, it could be named |
I like Branch |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Currently, the AOA/HID keyboard handles two different things:
- create the HID events
- send them over AOA
To reuse the HID part but using another protocol, your PR abstracts (2) (how to send them):
- introduce a new trait
sc_hid_interface
, implemented bysc_aoa
andsc_uhid
- an instance is passed to construct a
sc_hid_keyboard
This works, but it brings some complexity:
- two levels of interfaces implementations (something implementing a
sc_key_processor
, with one implementation receiving something implementing asc_hid_interface
) - the need for an
async_message
flag (because not all implementations behave exactly the same) - all implementation have to
create()
/destroy()
on the client-side (for UHID, I would prefer to create the UHID struct on the server side, the client should not have to handle all these details, likevendor_id
/product_id
, and the UHID handle should probably be closed on exit from the server side, without any message from the client)
I think "abstracting" the other way around ((1) extracting how to create the HID events) leads to a simpler solution (with less coupling).
My idea is to extract a component to create HID events (regardless of how they will be sent), so that separate keyboard implementations could reuse that same HID tools (without interfaces/traits) when they need to.
That way, we could get 3 separate keyboard/mouse "independant" implementations:
sc_keyboard_sdk
(previously `sc_keyboard_inject")sc_keyboard_aoa
(previouslysc_hid_keyboard
)sc_keyboard_uhid
Concretly:
app/src/hid
would contain HID toolsapp/src/usb
would contain the AOA keyboard/mouse implementationsapp/src/uhid
would contain the UHID keyboard/mouse implementations
I implemented preliminary commits which perform all these refactors with the current features: uhid.10
From this branch, you could more easily add a UHID implementation (independently of the AOA keyboard).
What do you think?
I agree with that. In fact, my app is using this architecture. |
How do I change the keyboard language? I already tried changing the language of the physical keyboard on Android but it still doesn't work. |
Here is my current branch: uhid.20. UHID and mod synchronization work. Please review and test. :) I also added a new shortcut MOD+k to open physical keyboard settings easily. Next steps:
|
I fixed several issues, and implemented mouse UHID: uhid.24 |
Note: on a Nexus 5 with Android 6, I get a permission denied when opening
$ adb shell ls /dev/uhid
/dev/uhid: Permission denied |
Oops, indeed some files were not committed. I forced pushed |
A few minor things I saw by looking quickly at the mouse implementation: diff1diff --git a/app/src/uhid/mouse_uhid.c b/app/src/uhid/mouse_uhid.c
index c3f78fd1..77446f9e 100644
--- a/app/src/uhid/mouse_uhid.c
+++ b/app/src/uhid/mouse_uhid.c
@@ -10,7 +10,7 @@
#define UHID_MOUSE_ID 2
static void
-sc_mouse_uhid_send_input(struct sc_mouse_uhid *kb,
+sc_mouse_uhid_send_input(struct sc_mouse_uhid *mouse,
const struct sc_hid_event *event, const char *name) {
struct sc_control_msg msg;
msg.type = SC_CONTROL_MSG_TYPE_UHID_INPUT;
@@ -20,7 +20,7 @@ sc_mouse_uhid_send_input(struct sc_mouse_uhid *kb,
memcpy(msg.uhid_input.data, event->data, event->size);
msg.uhid_input.size = event->size;
- if (!sc_controller_push_msg(kb->controller, &msg)) {
+ if (!sc_controller_push_msg(mouse->controller, &msg)) {
LOGE("Could not send UHID_INPUT message (%s)", name);
}
} diff2diff --git a/app/src/mouse_sdk.c b/app/src/mouse_sdk.c
index 620fb52c..37ba0736 100644
--- a/app/src/mouse_sdk.c
+++ b/app/src/mouse_sdk.c
@@ -57,7 +57,7 @@ convert_touch_action(enum sc_touch_action action) {
static void
sc_mouse_processor_process_mouse_motion(struct sc_mouse_processor *mp,
- const struct sc_mouse_motion_event *event) {
+ const struct sc_mouse_motion_event *event) {
if (!event->buttons_state) {
// Do not send motion events when no click is pressed
return;
@@ -83,7 +83,7 @@ sc_mouse_processor_process_mouse_motion(struct sc_mouse_processor *mp,
static void
sc_mouse_processor_process_mouse_click(struct sc_mouse_processor *mp,
- const struct sc_mouse_click_event *event) {
+ const struct sc_mouse_click_event *event) {
struct sc_mouse_sdk *m = DOWNCAST(mp);
struct sc_control_msg msg = {
@@ -105,7 +105,7 @@ sc_mouse_processor_process_mouse_click(struct sc_mouse_processor *mp,
static void
sc_mouse_processor_process_mouse_scroll(struct sc_mouse_processor *mp,
- const struct sc_mouse_scroll_event *event) {
+ const struct sc_mouse_scroll_event *event) {
struct sc_mouse_sdk *m = DOWNCAST(mp);
struct sc_control_msg msg = {
diff --git a/app/src/uhid/mouse_uhid.c b/app/src/uhid/mouse_uhid.c
index c3f78fd1..a4aa289b 100644
--- a/app/src/uhid/mouse_uhid.c
+++ b/app/src/uhid/mouse_uhid.c
@@ -27,7 +27,7 @@ sc_mouse_uhid_send_input(struct sc_mouse_uhid *kb,
static void
sc_mouse_processor_process_mouse_motion(struct sc_mouse_processor *mp,
- const struct sc_mouse_motion_event *event) {
+ const struct sc_mouse_motion_event *event) {
struct sc_mouse_uhid *mouse = DOWNCAST(mp);
struct sc_hid_event hid_event;
@@ -38,7 +38,7 @@ sc_mouse_processor_process_mouse_motion(struct sc_mouse_processor *mp,
static void
sc_mouse_processor_process_mouse_click(struct sc_mouse_processor *mp,
- const struct sc_mouse_click_event *event) {
+ const struct sc_mouse_click_event *event) {
struct sc_mouse_uhid *mouse = DOWNCAST(mp);
struct sc_hid_event hid_event;
@@ -49,7 +49,7 @@ sc_mouse_processor_process_mouse_click(struct sc_mouse_processor *mp,
static void
sc_mouse_processor_process_mouse_scroll(struct sc_mouse_processor *mp,
- const struct sc_mouse_scroll_event *event) {
+ const struct sc_mouse_scroll_event *event) {
struct sc_mouse_uhid *mouse = DOWNCAST(mp);
struct sc_hid_event hid_event;
diff --git a/app/src/usb/mouse_aoa.c b/app/src/usb/mouse_aoa.c
index 93b32328..adb0cfce 100644
--- a/app/src/usb/mouse_aoa.c
+++ b/app/src/usb/mouse_aoa.c
@@ -13,7 +13,7 @@
static void
sc_mouse_processor_process_mouse_motion(struct sc_mouse_processor *mp,
- const struct sc_mouse_motion_event *event) {
+ const struct sc_mouse_motion_event *event) {
struct sc_mouse_aoa *mouse = DOWNCAST(mp);
struct sc_hid_event hid_event;
@@ -27,7 +27,7 @@ sc_mouse_processor_process_mouse_motion(struct sc_mouse_processor *mp,
static void
sc_mouse_processor_process_mouse_click(struct sc_mouse_processor *mp,
- const struct sc_mouse_click_event *event) {
+ const struct sc_mouse_click_event *event) {
struct sc_mouse_aoa *mouse = DOWNCAST(mp);
struct sc_hid_event hid_event;
@@ -41,7 +41,7 @@ sc_mouse_processor_process_mouse_click(struct sc_mouse_processor *mp,
static void
sc_mouse_processor_process_mouse_scroll(struct sc_mouse_processor *mp,
- const struct sc_mouse_scroll_event *event) {
+ const struct sc_mouse_scroll_event *event) {
struct sc_mouse_aoa *mouse = DOWNCAST(mp);
struct sc_hid_event hid_event; |
Great job! This is a very nice solution. ;) A few years ago I implemented uinput & uhid injection on top of scrcpy for showing the mouse pointer, but I used JNI and didn't know that the mouse could be captured by SDL thus I used absolute motion on the client side and had to calibrate the pointer offset at the beginning on the server side (force the mouse outside the screen because its position gets clamped, then you can work with increments) so that both the pointers on the device and on the computer are aligned. I'm probably wrong but I'm wondering, maybe this could allow you to also handle touch events even though the mouse is in relative coordinates? I can clean up my code and share it with you if you wish to have a look. |
Thank you, fixed locally.
This is to not exceed 80 chars (that's maybe a stupid rule, but for consistency I stick to it).
I'm pretty sure it should be possible to simulate a HID touch screen (with absolute coordinates). To be investigated… |
Oh, sorry, that makes sense! |
So apparently it fails on Android 6. It works on Android 11. Could you please test on Android 7, 8, 9 and 10 to determine from which Android version it works? |
I can only confirm it works on Android 10.
|
Rename {keyboard,mouse}_inject to {keyboard,mouse}_sdk. All implementations "inject" key events and mouse events, what differs is the mechanism. For these implementations, the Android SDK API is used to inject events. Note that the input mode enum variants were already renamed (SC_KEYBOARD_INPUT_MODE_SDK and SC_MOUSE_INPUT_MODE_SDK). PR #4473 <#4473>
Use the following command: scrcpy --keyboard=uhid PR #4473 <#4473> Co-authored-by: Romain Vimont <[email protected]> Signed-off-by: Romain Vimont <[email protected]>
Use UHID output reports to synchronize CapsLock and VerrNum states. PR #4473 <#4473> Co-authored-by: Romain Vimont <[email protected]> Signed-off-by: Romain Vimont <[email protected]>
Initially, if AOA initialization failed, default injection method was used, in order to use the same command/shortcut when the device is connected via USB or via TCP/IP, without changing the arguments. Now that there are 3 keyboard modes, it seems unexpected to switch to another specific mode if AOA fails (and it is inconsistent). If the user explicitly requests AOA, then use AOA or fail. Refs #2632 comment <#2632 (comment)> PR #4473 <#4473>
|
Hi @yume-chan , Thanks for the update. I was just wondering if I could send a software message to scrcpy server like |
@chldbwnstm No, to factorize the code between AOA and UHID, the client sends the HID payloads, which are created by the client (but the UHID structure is created on the device side in the merged version). |
#4473 (comment) plus kernel version should be >= 3.15, see #4751 (comment). |
scrcpy v2.4 Changes since v2.3.1: - Add UHID keyboard and mouse support (Genymobile#4473) - Simulate tilt multitouch by pressing Shift (Genymobile#4529) - Add rotation support for non-default display (Genymobile#4698) - Improve audio player (Genymobile#4572) - Adapt to display API changes in Android 15 (Genymobile#4646, Genymobile#4656, Genymobile#4657) - Adapt audio workarounds to Android 14 (Genymobile#4492) - Fix clipboard for IQOO devices on Android 14 (Genymobile#4492, Genymobile#4589, Genymobile#4703) - Fix integer overflow for audio packet duration (Genymobile#4536) - Rework cleanup (Genymobile#4649) - Upgrade FFmpeg to 6.1.1 in Windows releases (Genymobile#4713) - Upgrade libusb to 1.0.27 in Windows releases (Genymobile#4713) - Various technical fixes
UHID may not work on old Android versions due to permission errors. Mention it in UHID mouse and gamepad documentation (it was already mentioned for UHID keyboard). Refs #4473 comment <#4473 (comment)> PR #5270 <#5270>
Fixes #4034
Fixes #3110
Fixes #3913
Currently only the keyboard part is implemented to get the overall architecture reviewed.
As described in #4034 (comment), the client is responsible for creating and parsing UHID messages, and the server only do the reads and writes to UHID file descriptors. This allows reusing existing AoA mouse/keyboard code to do the HID device emulation, and use native C code to handle UHID messages (which is also defined in C headers).
The new option is named
--keyboard-input-mode
, after its internal name. Available values aredisable
,inject
,aoa
anduhid
.aoa
deprecates the-K
/--hid-keyboard
option. When--otg
is not specified, all values are accepted, exceptaoa
doesn't work on Windows. When--otg
is specified, onlydisable
andaoa
are accepted. When--keyboard-input-mode
is not speficied, it defaults toinject
in normal mode andaoa
in OTG mode.Three new control messages are added:
UHID_OPEN
,UHID_WRITE
andUHID_CLOSE
. Each UHID device is a separate file descriptor, so all messages have anint id
field, generated by client, maps to a file descriptor on server.Currently
UHID_OPEN
andUHID_WRITE
both have abyte[] data
field (containing auhid_event
structure), so opening file and registering device can be done in one message. But this design is a little bit strange. Maybe we don't needUHID_OPEN
at all.One new device messages are added:
UHID_DATA
. It contains auhid_event
structure read from the file descriptor. In theory the client should wait for aUHID_DATA
messages withSTART
andOPEN
UHID message types before starting to send inputs, but in practice these two arrives pretty quickly, so when user starts typing, UHID already finished initializing.UHID_DATA
can also containOUTPUT
UHID messages, these are HID output reports defined in the device's report descriptor, for example LED status for keyboards and rumbles for gamepads. The code to handle these messages haven't been added to the client.