Overnight large-scale subjective video quality assessment using automatic test generation and crowdsourcing Internet marketplaces

Tamar Shoham; Dror Gill; Sharon Carmel; Dan Julius

doi:10.1117/12.2528012

6 September 2019 Overnight large-scale subjective video quality assessment using automatic test generation and crowdsourcing Internet marketplaces

Tamar Shoham, Dror Gill, Sharon Carmel, Dan Julius

Author Affiliations +

Proceedings Volume 11137, Applications of Digital Image Processing XLII; 111370R (2019) https://doi.org/10.1117/12.2528012
Event: SPIE Optical Engineering + Applications, 2019, San Diego, California, United States

Abstract

Subjective quality feedback by actual human viewers is crucial for reliable evaluation of various solutions and configurations for video processing or encoding. However, it is generally a time consuming and expensive process. Therefore, in many cases evaluation of video quality is done using objective measures, which may be poorly correlated with actual subjective results. In order to address this issue, we have developed VISTA (VIsual Subjective Testing Application), an easy to use application for visually comparing pairs of video sequences played synchronously side by side, and a user interface for indicating the relative subjective quality of the two video sequences. In addition, we developed a system for automating the evaluation process called Auto-VISTA. The system receives as input guidelines for the required testing session, prepares the content to be compared, launches the app in a crowdsourcing Internet marketplace (such as Amazon Mechanical Turk), and performs collection and analysis of the results. Thus, obtaining large scale subjective feedback becomes cheap and accessible, which in turn allows for fast and reliable evaluation cycles of different video encoding and processing solutions, or tuning various configurations and settings for a given solution.

Conference Presentation

Citation Download Citation

Tamar Shoham, Dror Gill, Sharon Carmel, and Dan Julius "Overnight large-scale subjective video quality assessment using automatic test generation and crowdsourcing Internet marketplaces", Proc. SPIE 11137, Applications of Digital Image Processing XLII, 111370R (6 September 2019); https://doi.org/10.1117/12.2528012

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available