®om's blog { un blog libre }

Introducing scrcpy

I developed an application to display and control Android devices connected on USB. It does not require any root access. It works on GNU/Linux, Windows and Mac OS.

scrcpy

It focuses on:

  • lightness (native, displays only the device screen)
  • performances (30~60fps)
  • quality (1920×1080 or above)
  • low latency (70~100ms)
  • low startup time (~1 second to display the first image)
  • non-intrusiveness (nothing is left installed on the device)

Like my previous project, gnirehtet, Genymobile accepted to open source it: scrcpy.

You can build, install and run it.

How does scrcpy work?

The application executes a server on the device. The client and the server communicate via a socket over an adb tunnel.

The server streams an H.264 video of the device screen. The client decodes the video frames and displays them.

The client captures input (keyboard and mouse) events, sends them to the server, which injects them to the device.

The documentation gives more details.

Here, I will detail several technical aspects of the application likely to interest developers.

Minimize latency

No buffering

It takes time to encode, transmit and decode the video stream. To minimize latency, we must avoid any additional delay.

For example, let’s stream the screen with screenrecord and play it with VLC:

adb exec-out screenrecord --output-format=h264 - | vlc - --demux h264

Initially, it works, but quickly the latency increases and frames are broken. The reason is that VLC associates a PTS to frames, and buffers the stream to play frames at some target time.

As a consequence, it sometimes prints such errors on stderr:

ES_OUT_SET_(GROUP_)PCR  is called too late (pts_delay increased to 300 ms)

Just before I started the project, Philippe, a colleague who played with WebRTC, advised me to “manually” decode (using FFmpeg) and render frames, to avoid any additional latency. This saved me from wasting time, it was the right solution.

Decoding the video stream to retrieve individual frames with FFmpeg is rather straightforward.

Skip frames

If, for any reason, the rendering is delayed, decoded frames are dropped so that scrcpy always displays the last decoded frame.

Note that this behavior may be changed with a configuration flag:

mesonconf x -Dskip_frames=false

Run a Java main on Android

Capturing the device screen requires some privileges, which are granted to shell.

It is possible to execute Java code as shell on Android, by invoking app_process from adb shell.

Hello, world!

Here is a simple Java application:

public class HelloWorld {
    public static void main(String... args) {
        System.out.println("Hello, world!");
    }
}

Let’s compile and dex it:

javac -source 1.7 -target 1.7 HelloWorld.java
"$ANDROID_HOME"/build-tools/27.0.2/dx \
    --dex --output classes.dex HelloWorld.class

Then, we push classes.dex to an Android device:

adb push classes.dex /data/local/tmp/

And execute it:

$ adb shell CLASSPATH=/data/local/tmp/classes.dex app_process / HelloWorld
Hello, world!

Access the Android framework

The application can access the Android framework at runtime.

For example, let’s use android.os.SystemClock:

import android.os.SystemClock;

public class HelloWorld {
    public static void main(String... args) {
        System.out.print("Hello,");
        SystemClock.sleep(1000);
        System.out.println(" world!");
    }
}

We link our class against android.jar:

javac -source 1.7 -target 1.7 \
    -cp "$ANDROID_HOME"/platforms/android-27/android.jar
    HelloWorld.java

Then run it as before.

Note that scrcpy also needs to access hidden methods from the framework. In that case, linking against android.jar is not sufficient, so it uses reflection.

Like an APK

The execution also works if classes.dex is embedded in a zip/jar:

jar cvf hello.jar classes.dex
adb push hello.jar /data/local/tmp/
adb shell CLASSPATH=/data/local/tmp/hello.jar app_process / HelloWorld

You know an example of a zip containing classes.dex? An APK!

Therefore, it works for any installed APK containing a class with a main method:

$ adb install myapp.apk
…
$ adb shell pm path my.app.package
package:/data/app/my.app.package-1/base.apk
$ adb shell CLASSPATH=/data/app/my.app.package-1/base.apk \
    app_process / HelloWorld

In scrcpy

To simplify the build system, I decided to build the server as an APK using gradle, even if it’s not a real Android application: gradle provides tasks for running tests, checking style, etc.

Invoked that way, the server is authorized to capture the device screen.

Improve startup time

Quick installation

Nothing is required to be installed on the device by the user: at startup, the client is responsible for executing the server on the device.

We saw that we can execute the main method of the server from an APK either:

  • installed, or
  • pushed to /data/local/tmp.

Which one to choose?

$ time adb install server.apk
…
real    0m0,963s
…

$ time adb push server.apk /data/local/tmp/
…
real    0m0,022s
…

So I decided to push.

Note that /data/local/tmp is readable and writable by shell, but not world-writable, so a malicious application may not replace the server just before the client executes it.

Parallelization

If you executed the Hello, world! in the previous section, you may have noticed that running app_process takes some time: Hello, World! is not printed before some delay (between 0.5 and 1 second).

In the client, initializing SDL also takes some time.

Therefore, these initialization steps have been parallelized.

Clean up the device

After usage, we want to remove the server (/data/local/tmp/scrcpy-server.jar) from the device.

We could remove it on exit, but then, it would be left on device disconnection.

Instead, once the server is opened by app_process, scrcpy unlinks (rm) it. Thus, the file is present only for less than 1 second (it is removed even before the screen is displayed).

The file itself (not its name) is actually removed when the last associated open file descriptor is closed (at the latest, when app_process dies).

Handle text input

Handling input received from a keyboard is more complicated than I thought.

Events

There are 2 kinds of “keyboard” events:

Key events provide both the scancode (the physical location of a key on the keyboard) and the keycode (which depends on the keyboard layout). Only keycodes are used by scrcpy (it doesn’t need the location of physical keys).

However, key events are not sufficient to handle text input:

Sometimes it can take multiple key presses to produce a character. Sometimes a single key press can produce multiple characters.

Even simple characters may not be handled easily with key events, since they depend on the layout. For example, on a French keyboard, typing . (dot) generates Shift+;.

Therefore, scrcpy forwards key events to the device only for a limited set of keys. The remaining are handled by text input events.

Inject text

On the Android side, we may not inject text directly (injecting a KeyEvent created by the relevant constructor does not work). Instead, we can retrieve a list of KeyEvents to generate for a char[], using getEvents(char[]).

For example:

char[] chars = {'?'};
KeyEvent[] events = charMap.getEvents(chars);

Here, events is initialized with an array of 4 events:

  1. press KEYCODE_SHIFT_LEFT
  2. press KEYCODE_SLASH
  3. release KEYCODE_SLASH
  4. release KEYCODE_SHIFT_LEFT

Injecting those events correctly generates the char '?'.

Handle accented characters

Unfortunately, the previous method only works for ASCII characters:

char[] chars = {'é'};
KeyEvent[] events = charMap.getEvents(chars);
// events is null!!!

I first thought there was no way to inject such events from there, until I discussed with Philippe (yes, the same as earlier), who knew the solution: it works when we decompose the characters using combining diacritical dead key characters.

Concretely, instead of injecting "é", we inject "\u0301e":

char[] chars = {'\u0301', 'e'};
KeyEvent[] events = charMap.getEvents(chars);
// now, there are events

Therefore, to support accented characters, scrcpy attempts to decompose the characters using KeyComposition.

EDIT: Accented characters do not work with the virtual keyboard Gboard (the default Google keyboard), but work with the default (AOSP) keyboard and SwiftKey.

Set a window icon

The application window may have an icon, used in the title bar (for some desktop environments) and/or in the desktop taskbar.

The window icon must be set from an SDL_Surface by SDL_SetWindowIcon. Creating the surface with the icon content is up to the developer. For exemple, we could decide to load the icon from a PNG file, or directly from its raw pixels in memory.

Instead, another colleague, Aurélien, suggested I use the XPM image format, which is also a valid C source code: icon.xpm.

Note that the image is not the content of the variable icon_xpm declared in icon.xpm: it’s the whole file! Thus, icon.xpm may be both directly opened in Gimp and included in C source code:

#include "icon.xpm"

As a benefit, we directly “recognize” the icon from the source code, and we can patch it easily: in debug mode, the icon color is changed.

Conclusion

Developing this project was an awesome and motivating experience. I’ve learned a lot (I never used SDL or libav/FFmpeg before).

The resulting application works better than I initially expected, and I’m happy to have been able to open source it.

Discuss on reddit and Hacker News.

Comments

Cascador

Salute,

Tu t’en sers pour faire quoi ? Quels sont les cas d’usage ?

Merci, Tcho !

®om

@Cascador :

I cross-post my answer (in English):

It has been developed for a specific use case (it is included in a B2B application where the user may need to display and control devices connected on USB).

But you might use it for other reasons, like typing text messages from your computer, or showing your device in a conference (e.g. to demo an app).

https://news.ycombinator.com/item?id=16546025

Tcho ! ;-)

Hola,

This is such great piece of work, thanks for sharing!

Cascador

Salute,

Tu comptes publier un article en Français ? Je l’aurai bien remonté sur le Journal du hacker. Pour envoyer des SMS c’est très simple/pratique ou il faut avoir envie et faire des efforts selon toi ?

Thanks ha ha, Tcho !

®om

J’avais hésité, mais finalement je ne vais pas publier en français: ça demande trop de travail d’écrire les articles en double.

Je ne suis pas sûr d’avoir compris ta question sur les SMS. Moi, je trouve plus pratique de les écrire avec le clavier du PC.

onioncoder

This is awesome. Love that it is a cross platform native app :)

dro

Could you make it run on older android versions like 4.3? Also can you please share your app on f-droid too?

@cascador : utilises plutôt kconnect pour récupérer les sms vers ton ordi. Tu peux même faire du cross ^C^V !

rayworks

A great project!About one year ago, I just made a demo app to take a screenshot dynamically. However,the result you achieved is what I actually want.

Terpe

Thanks! Learned new thinks

j-gatsby

Awesome project. Excellent write-up.

And thank you for sharing it with us.

Great work! Very well done.

Trevor Parsons

Thanks for writing and publishing scrcpy. It was easy to install via AUR on my Arch-based desktop and it works perfectly. Much appreciated!

jg

This looks amazing! I tried it on Windows with my V20 and the image displays immediately and updates quick but it does not respond to my clicks. :(

®om

@jg, it seems that clicks do not work with LG devices on this version.

Louie

What version of Windows is compatible? It does not work on Windows 10.

Hoang

Thanks for the app. Looks good and fast, but I can’t get edge-swiping to work, which limits usability quite a bit. Likewise no pinch-zoom. Hope to see these in a future iteration.

Seb

@Louie, I don’t know if it’s specific to Windows 10 but the first time I started the app it appeared not to work either. In my case the debugging mode trust dialog appeared on my phone but nothing happened after allowing the connection. Executing the app again worked but it spawned a duplicate process on my computer (CTRL+ALT+DEL -> Task Manager – didn’t know which one was the good one so I killed both). It appears to be working fine now.

@Romain, nice work! Thanks for opening the source! I haven’t checked the code yet but maybe there’s an issue when connecting the phone in debugging mode for the first time on an untrusted computer?

®om

@Seb, Thank you for reporting the problem. I didn’t check yet. Please open an issue on GitHub.

just_found

Works great! Ran error free.

@Louie…using win10.

I have been looking for something like this for a while.

Thank you for contributing to the open source community.

Seb

@Romain,

It seems like the issue (or at least a very similar one) has already been submitted (Issue #9). I’ve replicated the problem on another computer, so I’ve added this info as a comment to the existing issue.

Thanks!

claude

On Windows 7 32 bit. it doesn’t work.

Jakob

Thanks, that’s great! Any chance this would work in 1280x720 screens ?

Great work! Have you thought about mentioning it on https://www.xda-developers.com/ ?

Other thing. I’ve learnt about haven’t and (forgive me) I haven’t tried it your app yet, but from:

It is possible to execute Java code as shell on Android, by invoking app_process from adb shell.

and given that it’s possible to run adb over the network (adb connect device over TCP port 5555), I’m inclined to think that it would possible to run your app without having the droid physically connected over USB, which in turn would make your app a really good remote support app. If you see what I mean.

Dominique

Harpo

Whoa, if it supported audio, and coupled with a Raspberry Pi and a touchscreen, this would make a killer car head unit!

I’m sick of Android Auto being a walled garden and only the Google-vetted apps being available.

Jermaine

Works great !

Sinan Arslan

Could you please prepare a video tutorial about downloading and running Scrcpy on Windows? Your blog is helpful but yet not so clear.

oldboy

Did not work with my Asus phone for more than 10 seconds max but with a Sony it works pretty well. The computer is Raspberry Pi 3B. Nice work!

Manish

First of all, a huge thanks to the developer for this solution. I’ve been looking for something like this for a while. Most of the solutions i found had a substantial lag(my 2nd Pentium made it worse). But this works smoothest and startup is fast too…Thanks again!

One thing I’m unable to figure out is, how to use navigation keys(Back,Home,Recents) in scrcpy, as my device have H/w keys

Juan

Congratulations for your creation. I have been looking for something similar (vysor, samsung sidesync) but nothing compares to this in simplicity and speed. Nothing (only adb drivers) to install on computer and phone and it works, that’s what I call sleek.

The new mouse shorcuts are very convenient, I like them! And for file transfer I can just use the plain usb connection.

Regarding issues found: keyboard is also working with samsung keyboard. Seems something google keyboard specific. Also vol+ and vol- keyboard shorcuts doesn’t work (on lineageos 14/samsung nougat)

Keep up the good work! My whishlist: Audio routed to the computer. I’m unable to help but I know that android has a mode where it behaves as a player that can route audio back through usb and get play/pause/stop commands. It is called (I believe) accessory mode https://source.android.com/devices/audio/usb

®om

@Juan Thank you ;)

Also vol+ and vol- keyboard shorcuts doesn’t work

Please open an issue.

It is called (I believe) accessory mode https://source.android.com/devices/audio/usb

Great suggestion! Thank you.

You can watch issue 14 for audio support.

Super projet qui m’a aussi fait découvrir par la même occasion, un blog hyper intéressant :)

Bon courage pour la suite, j’attends de nouveaux articles sur ton blog avec impatience.

PS: c’est drôle le hasard quand même, j’ai vu qu’on venait de la même école d’ingénieurs haha!

Cihan Mercan

Great program. Great work. Thanks for sharing it with us.

Matthew B

I’ve had a chance to try out scrcpy and I like it quite a lot. Do you know if it is feasible to have the devices screen off while streaming the video to the desk? I feel like this might not be possible but it would be nice for conserving the device battery.

®om

Do you know if it is feasible to have the devices screen off while streaming the video to the desk?

IMO, turning device screen off is not possible.

You could change the brightness, though (using a wrapper script).

justinbacle

Hi, Love this stuff ! Just wanted to report that on MIUI 9.2, you need to enable both “USB debugging” and “USB debugging (Security Settings)” for it to work ;)

Alex

What I’ve been using since more than 6 years is the application WifiKeyboard :-) that’s also a nice alternative!

abe

Hello dev. I have tried the audio branch. Great work. Please make it work over adb connect. Many thanks for the project

weak-eyes

Hi, Nice piece of software. I need it magnified on my linux PC but i cannot magnify it. Can you add a magnification feature to scrcpy. Thank-you.

Rahul

Hi,

I am getting an error while trying out your HelloWorld.java program with sleep. Its throwing an error while executing it the system.

Exception in thread "main" java.lang.UnsatisfiedLinkError: No implementation found for long android.os.SystemClock.uptimeMillis() (tried Java_android_os_SystemClock_uptimeMillis and Java_android_os_SystemClock_uptimeMillis__)
        at android.os.SystemClock.uptimeMillis(Native Method)
        at android.os.SystemClock.sleep(SystemClock.java:117)
        at HelloWorld.main(HelloWorld.java:9)
Wikotz

Absolument génial! Le fonctionnement est impeccable sur un Motorola E4 (android 7.1.1).

J’ai connu une seule question-qui consiste à “glisser” lorsque l’écran est tombé en veille.

Serait-il possible de faire appel à “glisser” à l’aide du clavier (ex: CTRL+u ou F11)?

Merci et meilleurs voeux!


Absolutely brilliant! Operation is flawless on a Motorola E4 (android 7.1.1).

I have experienced only one issue – which involves “swiping up” when the screen has dropped into standby.

Would it be possible to call “swiping up” via the keyboard (ex: CTL+u or F11)?

Thank you and best wishes!

AKHIL THAKUR

great work man …….. working smoothly on android 5.0 and above ,but not working on my kikat 4.4 . please make an update for this..

Matt h

Thank you so much for developing this app and giving it to the world. Works so well. I’m actually blown away. For me on my OnePlus 3T, scrcpy works as well as DisplayLink. Simply incredible. Again thanks for your time and effort in developing this app.

Elliot Nathanson

This is outstanding. Can’t thank you enough. FWIW, I hacked up a script to find an open port so I can connect scrcpy to multiple devices simultaneously.

#!/bin/bash

# find unused port
for PORT in $(seq 27183 65535); do
  COUNT=$(nmap localhost -PS -p $PORT -oG - | grep -c "closed")

  if [ $COUNT -ge 1 ]; then
    break
  fi

done

scrcpy -p $PORT &
®om

FWIW, I hacked up a script to find an open port so I can connect scrcpy to multiple devices simultaneously.

FYI, scrcpy should be listening on the port for about 1 second, so unless you want to start several devices in the same second, you should be able to run several scrcpy instances without using a custom port.

Przemo-c

Is there any chance of getting an android app of scrcpy to have android->android mirror?

I’m using it to mirror Oculus Go on PC and it works great both wired and wireless.

But it would be great to have it mirrored to my smartphone.

Proximo

I’ve been looking for this feature as well: android->android mirror.

I want to be able to control one smart phone from another over usb because the wireless traffic is unwanted.

Thank you for this well-documented and clean program.

Kris

I went looking for a replacement to SideSync on my S9, since it isn’t supported and Flow is terrible by comparison. I stumbled upon this and it is fantastic!

There are a couple of things that I still miss like the notifications from my phone on my PC, easy file transferring, automatic screen dimming or turning off the screen completely, etc., but it is still much, much better than Flow and other mirroring options.

If anyone is interested, I used Tasker to turn the brightness to zero and keep the phone awake when charging when charging via USB (i.e. connected to my computer) and near my computer’s bluetooth address. My S9 using scrcpy actually charges faster than my S7 did using SideSync, even though SideSync was able to completely turn the screen off!

Dolly

Nice app. Thank you for this (and for gnirehtet too, using it daily!). Would be better if there was a way to add navigation bar though. I’m not really into keyboard shortcuts

®om

@Dolly Note that BACK and HOME are also bound respectively to right- and middle-click.

HappyUser666

Great work, super fast and easy for sharing an Android Screen, or every day use (I use it at work to receive and send messages, while charging my phone). The only need is an usb cable

Pinch zoom support (ctrl + mouse wheel ?) would be a killer feature.


French version : Super boulot, super rapide et facile de partager un écran d’un smartphone Android, ou pour une utilisation quotidienne (Je l’utilise au travail pour mes sms, mes recherches persos, et en même temps mon tél se recharge, le pied). Il faut juste un cable USB

Le support du zoom / dézoom avec par exemple ctrl+molette de la souris serait une fonctionnalité de la mort (désolé pour la trad de “killer feature” qui fait djeun’s des années 1990)

AdotK

Hello, i am thankful for scrcpy. I can control almost every application except one : Grim defender android game. Somehow this game don’t register any mouse clicks and keystouch. Seem like a pretty tight security any help?

Antonio Bray

Thanks for developing this. How hardwould it be to develop an Android client? Also what would help increase low latency? GPU assisted encoding or use of H265.xx streaming?

Is this type of performance possible?

https://lunadisplay.com/

https://www.youtube.com/watch?v=PDd0DT-5r20

Thanks

Nils

Hi !

GREAT stuff, thank you ! Do you know if there is an Android client for scrcpy ? So i could mirror one Android device to another (via wifi ADB) ?

Greetings,

Nils

®om

@HappyUser666 See #24.

@AdotK Please open an issue with more details (device, orientation, result of adb logcat…). Also check open input events issues.

@AntinioBray The main latency is due to encoding/decoding. Thus, to improve it, you need to encode and decode faster. One possibility is simply to reduce the definition (scrcpy -m800).

@Nils See #100/scrcpy-android.

Nils

Thanks a lot ®om !

I tried the android scrcpy and it is exactly what i need but it did not work up to now but i will try and ask that developer.

THANKS for pointing me there, you’re way better then google

Nils

daddy366

Built scrcpy and every thing works great on my Nexus 6p,tried it on my motorola e5 plus and its a no go. This is what i get running it.

/usr/local/share/scrcpy/scrcpy-server....shed. 2.5 MB/s (18570 bytes in 0.007s)
Aborted

Would love to get it working on the moto…Great work Thank you

Post a comment

Name : (required)
E-mail : (optional, non published)
Website : (optional)
What is the 3rd letter of the word blog? (antispam)

Comments are formatted in markdown.