A practical approach for optical skimming and automatic identification of the pressed contents on touchscreen devices (Conference Presentation)

Dinh Khoi Le; Volker Krummel; Moritz Blum; Benedikt J. Lueke

doi:10.1117/12.2325347

11 October 2018 A practical approach for optical skimming and automatic identification of the pressed contents on touchscreen devices (Conference Presentation)

Dinh Khoi Le, Volker Krummel, Moritz Blum, Benedikt J. Lueke

Author Affiliations +

Proceedings Volume 10802, Counterterrorism, Crime Fighting, Forensics, and Surveillance Technologies II; 108020P (2018) https://doi.org/10.1117/12.2325347
Event: SPIE Security + Defence, 2018, Berlin, Germany

Abstract

Entering confidential information is the classical application for optical skimming. For example entering a PIN or password to authenticate banking transactions is one of the most sensitive moments of a transaction. Up to now it was unclear how efficient optical skimming attacks can be mounted even on mobile devices. In this paper we show that filming the entering of a password with the camera of a standard mobile phone is enough for a fully automated recovering of the sensitive information. Our analysis method of the recorded video stream leads to a success rate of more than 90%. In our model a user enters his password into an Android or IOS driven touchscreen device while being filmed by the attacker’s smartphone. The goal of the attacker is to derive the password from the movie instantly, e.g. to use it in a real time man-in-the-middle attack. On first sight such an attack seems hard to mount due to many disturbing factors like movements of the device, bad light conditions etc. However, we show that many of these disturbing factors can be mitigated by smart video analytics. We implemented the whole attack in our lab simulating real conditions. Parts of the setup where a Samsung S7 for filming and an iPhone 6 as target. For real time processing we use the computer vision library OpenCV which supports most of the common image and video processing algorithms. Our goal is to cover most of the important cases while holding as little knowledge as possible about the video the algorithm has to work on. Starting from movable or instable position for camera and device we apply a discrete Fourier transformation to obtain a constant plan view as starting point for our algorithm. To obtain the necessary information for this the user has to select the four corners of the keyboard which is observed by hand. In real live situations the device is hold by hand of the user so we decided to use the MOSSE algorithm for stabilization to deal with the problem of a jiggling device. The keystroke detection is based on the “pop up” feature of common smartphone keyboards. This event is remarkable enough to get detected and located by simple gray value subtraction of following frames. Not only the magnified keys are the primary changes we observed and analyzed. Minor changes like noise can be removed by sufficiently blurring the frame. Subtracting areas where normal skin colors are detected yields information about the position of the keystrokes. The relative position obtains a probability for the pressed keys by analyzing the activity of the regions where the “pop ups” usually are. Our work showed that optical skimming of passwords are practical even on mobile devices. We show that it is possible to make a simple attack with basic computer vision techniques. Up to date mobile devices have enough computing power to provide for this kind of attacks. We give recommendations for entering passwords securely.

Conference Presentation

Citation Download Citation

Dinh Khoi Le, Volker Krummel, Moritz Blum, and Benedikt J. Lueke "A practical approach for optical skimming and automatic identification of the pressed contents on touchscreen devices (Conference Presentation)", Proc. SPIE 10802, Counterterrorism, Crime Fighting, Forensics, and Surveillance Technologies II, 108020P (11 October 2018); https://doi.org/10.1117/12.2325347

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available

Members: $17.00

Non-members: $21.00 ADD TO CART

PROCEEDINGS
PRESENTATION

WATCH
PRESENTATION SAVE TO MY LIBRARY

GET CITATION

KEYWORDS

Mobile devices

Video

Cameras

Computer vision technology

Detection and tracking algorithms

Image processing

Machine vision

Show All Keywords

Keywords/Phrases

Search In:

Publication Years