Widget detection on screenshots using computer vision and machine learning algorithms

Kacper Radzikowski; Karol Chęciński; Mateusz Forc; Łukasz Lepak; Michał Jabłoński; Wiktor Kuśmirek; Bartłomiej Twardowski; Paweł Wawrzyński; Robert M. Nowak

doi:10.1117/12.2536406

6 November 2019 Widget detection on screenshots using computer vision and machine learning algorithms

Kacper Radzikowski, Karol Chęciński, Mateusz Forc, Łukasz Lepak, Michał Jabłoński, Wiktor Kuśmirek, Bartłomiej Twardowski, Paweł Wawrzyński, Robert M. Nowak

Author Affiliations +

Proceedings Volume 11176, Photonics Applications in Astronomy, Communications, Industry, and High-Energy Physics Experiments 2019; 111761Q (2019) https://doi.org/10.1117/12.2536406
Event: Photonics Applications in Astronomy, Communications, Industry, and High-Energy Physics Experiments 2019, 2019, Wilga, Poland

Abstract

In this paper we consider the problem of detecting and recognizing widgets in screenshots of computer programs’ graphical user interface (GUI). This problem is fundamental in business process automation. The solution we propose here is based on detecting GUI elements with Canny edge operator, and recognizing already detected GUI elements with classifiers: neural networks, random forests, XGBoost, and others.

Citation Download Citation

Kacper Radzikowski, Karol Chęciński, Mateusz Forc, Łukasz Lepak, Michał Jabłoński, Wiktor Kuśmirek, Bartłomiej Twardowski, Paweł Wawrzyński, and Robert M. Nowak "Widget detection on screenshots using computer vision and machine learning algorithms", Proc. SPIE 11176, Photonics Applications in Astronomy, Communications, Industry, and High-Energy Physics Experiments 2019, 111761Q (6 November 2019); https://doi.org/10.1117/12.2536406

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available