Double input along with latin accents

[dstwwy]

Hello.

I'm using the suggested keyboard app in #431 on my main phone and the stock version of LineageOS keyboard on my secondary phone, and both are able to type accented (áèîõü) characters as explained in https://blog.rom1v.com/2018/03/introducing-scrcpy/#handle-text-input.

However, when inputing [accent] then [key], scrcpy isn't able to recognize that I want a accented character only, so áá turns into aáaá. Is it possible to filter the output when I type an accent, like by just skiping the first output?

Thanks for the project, it was a lifechanger ever since its first release.

[rom1v]

To fix #87, since v1.4, scrcpy handles letters differently than other text (411aa4f). This is the cause of this bug.

I should probably manage several "input modes" and introduce a parameter to select it, because I think there is no solution working reliably for all cases. The modes would be:

• raw key events only (ignore text events)
• text events (ignore raw key events which produce text) (this was th behavior until 1.3, the one you want)
• mixed (text events only for non-letters) (this is the current behavior, a compromise to fix Unable to use keyboard on Unreal Engine game #87 while working relatively good for text)

[dstwwy]

Started testing 1.3 and got a bunch of wildly varying results, before noticing that I was on Gboard. Thanks Google.

So far it behaves as you said on 1.3, so I'll use both according to my needs. As for your solution, it seems that every options brings issues, so I don't know if it's worth pursuing instead of waiting for a fix on how adb works (dunno if even possible) or something else.

[rom1v]

This is not a bug in adb, just a conflict between injecting text and injecting press+release events.

[dstwwy]

It's not caused by how adb works (#214)? Although you explained how you are handling accents on your blog, so you might be sidestepping this issue. If so I'll be here for tests, if needed.

[rom1v]

It's not caused by how adb works (#214)?

No. There are 2 issues:

1. Android only supports ASCII for injecting text.
2. we need different events for generating text or for pressing keys.

If you inject à, with the workaround explained in the blog post, we can inject the text correctly, so the first issue is not a problem here.

In itself, here, the second issue should not be a problem either: it is clear that we want to inject à as text.

But related to #87, when we press a letter, sometimes we want a letter (because we are typing text), sometimes we want the press+release events (typically wasd in a game). So to support this use case, all text events with letters (without accents) are ignored and raw events for letters are forwarded instead.

As a consequence, now when you press à, it generates both the "key a pressed", "key a released", and "text à".

In other words, even if issue 1 did not exist, you would still have the problem.

If so I'll be here for tests, if needed.

👍

[dstwwy]

Ah, a key up event, the game example was very clear. Now I see the problem.

[dstwwy]

Could you add your parameter suggestion on a new release, while there's no definitive solution? Doesn't look like it should break anything since the legacy code would stay inactive unless selected.

[rom1v]

Yes, I planned to add an option --text-events={all, non-alpha, none} before the v1.10 release, but I didn't had time to implement it. So I'll do it later :)

[fefo-dev]

Hi @rom1v , do you still plan to add the toggle on your next release? So far I've been using both 1.3 and 1.9 (.10 has the console issue), and I'm looking forward to it.

[rom1v]

do you still plan to add the toggle on your next release?

Yes.

For 1.11, in addition to what is already on dev, I would like to fix:

• Touch input offset on secondary monitor #15 (PR fix for resizing HIDPI MacOS windows #848)
• scrcpy-noconsole.exe shows a console #691 (no-console issue)
• this one

[rom1v]

I implemented it in textevents.

scrcpy-textevents.zip
SHA256: 0086a66ca8eac3faef21082becd696b33cd4c0e803c8acd6ba1d8ec068d8e1a9

(extract this in your scrcpy v1.10 folder to override)

Now, the text works like in v1.3:

scrcpy --prefer-text-events=always  # this is the default

To get the behavior of v1.4→v1.10:

scrcpy --prefer-text-events=non-alpha

And there is another mode, which never sends text events (probably useful only for US keyboard on both the computer and the device).

scrcpy --prefer-text-events=never

[rom1v]

I think I will simplify, and only keep the behavior of always and non-alpha, so that the option may become --prefer-text-events or even --prefer-text without parameter.

[fefo-dev]

Working as intended here on default, thanks. Dunno how optimal this solution works for you, but I can't see a better way to do it outside of a toggle on an UI that doesn't exist now.

If you got a no console build I wouldn't mind, hehe. Bundling it on the main app is a bigger headache that doing both?

[rom1v]

If you got a no console build I wouldn't mind

#691 is not fixed yet, so the scrcpy-noconsole.exe is still broken.

I will probably keep the v1.10 behavior by default (and --prefer-text will enable the option).

[rom1v]

I think I will simplify, and only keep the behavior of always and non-alpha, so that the option may become --prefer-text-events or even --prefer-text without parameter.

Done: c916af0 (textevents.2)

[rickx]

Testing textevents.2 but i only get the stream and 1 chance for 1 (mouse) input. After that it does not pass any input at all. Win64 version.

[rom1v]

@rickx Did you also recompile the server part?

You should get an error in the ouput of adb logcat when it start to fail, please check.

Could you test dev branch (the commit is now merged into dev).

[rickx]

hmm nope, just copied the server. I should compile it, right?

That's what I get:
11-15 11:30:44.469 593 755 E InputDispatcher: Key event has invalid action code 0x3
11-15 11:30:44.470 593 755 W InputManager: Input event injection from pid 26968 failed.

(twice).
Then it stops.

[rickx]

On dev now. Server compilation gives me error so I'm stuck.

ninja -C "build-server"
ninja: Entering directory `build-server'
[0/1] Generating scrcpy-server with a custom command.

> Task :server:compileReleaseAidl FAILED
/usr/lib/android-sdk/build-tools/28.0.3/aidl: 1: /usr/lib/android-sdk/build-tools/28.0.3/aidl: Syntax error: redirection unexpected


FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':server:compileReleaseAidl'.
> com.android.ide.common.workers.WorkerExecutorException: 1 exception was raised by workers:
  java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException: com.android.ide.common.process.ProcessException: Error while executing process /usr/lib/android-sdk/build-tools/28.0.3/aidl with arguments {-p/usr/lib/android-sdk/platforms/android-29/framework.aidl -o/home/federico/scrcpy_dev/scrcpy/server/build/generated/aidl_source_output_dir/release/compileReleaseAidl/out -I/home/federico/scrcpy_dev/scrcpy/server/src/release/aidl -I/home/federico/scrcpy_dev/scrcpy/server/src/main/aidl -d/tmp/aidl215713742388453911.d /home/federico/scrcpy_dev/scrcpy/server/src/main/aidl/android/view/IRotationWatcher.aidl}

[rom1v]

hmm nope, just copied the server. I should compile it, right?

Yes, that's the problem.

11-15 11:30:44.469 593 755 E InputDispatcher: Key event has invalid action code 0x3

This is due to a protocol issue between the client and the server ("they don't talk the same language").

[rom1v]

How did you install your Android SDK?

/usr/lib/android-sdk/build-tools/28.0.3/aidl: 1: /usr/lib/android-sdk/build-tools/28.0.3/aidl: Syntax error: redirection unexpected

It seems related to SDK build tools 28.0.3: #22 (comment)

Could you upgrade to 29.*?

[rickx]

I found out it's not possible because I was on a x386 system. Restarted with buster 64 bit. After some trouble with java I was successful, everything built fine.

[rickx]

Still issues with input. I'd like to have the debug output so I added those log entries in scrcpy.c. But I get no output in cmd window (w10). Do I need to set a debug flag or something?

[rom1v]

Still issues with input.

Which issues? The same?

But I get no output in cmd window (w10).

How did you build it?

Do I need to set a debug flag or something?

By default, if you don't pass --buildtype=release to the meson command, it's in debug.

[rickx]

Still issues with input.

Which issues? The same?

Well, it's something weird. It's most probably not scrcpy's fault. I have an android app written in qt. This app has some input fields which "give a twist" to the input, when the input is done through scrcpy: usually they take the first letter and the last and switch them. So e.g. HELLO becomes ELLOH.
This only happens when the input field looses focus, it looks good before that. If I use the onscreen keyboard which pops up (android's), the twist does not happen. So, to debug this, I wanted to activate the logging of what is sent from pc to device.

But I get no output in cmd window (w10).

How did you build it?

Just make -f Makefile.CrossWindows

Do I need to set a debug flag or something?

By default, if you don't pass --buildtype=release to the meson command, it's in debug.

ok, that's probably the thing. Thanks. Does that mean I just remove the buildtype param? Or replace release with something else?

[rom1v]

I have an android app written in qt. This app has some input fields which "give a twist" to the input, when the input is done through scrcpy: usually they take the first letter and the last and switch them. So e.g. HELLO becomes ELLOH.

Can you reproduce it by injecting the text events manually (without scrcpy):

adb shell input text H
adb shell input text E
adb shell input text L
adb shell input text L
adb shell input text O

Does that mean I just remove the buildtype param? Or replace release with something else?

Either remove or replace by --buildtype=debug (you need to remove your builddir beforehand; alternatively, you can change its configuation using meson configure your_builddir --buildtype=debug).

[alexdevos]

Hello @rom1v, I'm running scrcpy version 1.12 (installed via snap) and I'm unable to have character as é, à or è when I type those character on my laptop keyboard : nothing appears on my Android phone. I tried scrcpy --prefer-text but I have same problem.
When I try adb shell input text E in a command line, I get the E on the phone, but with adb shell input text é nothing appears.
I also read #431 but it didn't help me. What can I do/test ?

By the way, thank you for this great tool 😃 !

[rom1v]

What is your keyboard on Android?

[alexdevos]

What is your keyboard on Android?

I have AnySoftKeyboard French v. 4.0.925.2525 (on AnySoftKeyboard v. 1.10.1096.6266) but I also tried with the LineageOS 16 stock keyboard and it doesn't work (I tried on two different phones with LOS 16 : Samsung Galaxy S5 and Samsung Galaxy S5+).

edit : I have also the same problem with Samsung Galaxy S3 and LineageOS 14.1 stock keyboard

[KaKi87]

Hello,

While using --prefer-text, there are still some special characters that I can't input : ², ¨, £, ¤, µ, §

Thanks

[rom1v]

@KaKi87 https://github.com/Genymobile/scrcpy/blob/master/FAQ.md#special-characters-do-not-work

[KaKi87]

Injecting text input is limited to ASCII characters.

• ² is ASCII 253
• ¨ is ASCII 249
• £ is ASCII 156
• ¤ is ASCII 207
• µ is ASCII 230
• § is ASCII 245

[rom1v]

ASCII is on 7 bits, so it contains only values up to 127.

I don't know what are the values you provide, since they are not latin1/cp1252/iso8859-1 either.

[KaKi87]

I see, so it only supports the non-extended ASCII table.

https://theasciicode.com.ar/

Would it be possible to implement support for the rest ?

[rom1v]

This is a limitation on Android on key injection. There are alternatives (like installing a separate app containing IME), see #37 and related.

[Kajziba]

Ah, a key up event, the game example was very clear. Now I see the problem.