Quality assessment of higher resolution images and videos with remote testing

TU Ilmenau / AVT

Der Artikel diskutiert die Verwendung von Nutzerkommentaren bei der Bewertung der Qualität von Multimediadaten und die Herausforderungen, die mit der Beschaffung solcher Kommentare durch Labortests verbunden sind. Um die hohen Kosten und den Zeitaufwand von Labortests zu reduzieren, schlägt der Artikel die Verwendung von Online-Tests vor. Der Artikel stellt drei verschiedene Online-Tests vor, die entwickelt wurden, um die visuelle Qualität von hochauflösenden Bildern und Videos mit Hilfe der Open Source Online-Testplattform AVrate Voyager zu bewerten. Es wird diskutiert, inwieweit Online-Tests ein zuverlässiger Ersatz für Labortests sein können. Betrachtet werden auch die möglichen Einschränkungen in Hinblick auf Anzeigegeräte, Webtechnologien, Videocodecs und -formate.

Steve Göring, Rakesh Rao Ramachandra Rao, Alexander Raake
Quality assessment of higher resolution images and videos with remote testing
Journal Quality and User Experience
DOI 10.1007/s41233-023-00055-6

View and download PDF

Abstract

In many research fields, human-annotated data plays an important role as it is used to accomplish a multitude of tasks. One such example is in the field of multimedia quality assessment where subjective annotations can be used to train or evaluate quality prediction models. Lab-based tests could be one approach to get such quality annotations. They are usually performed in well-defined and controlled environments to ensure high reliability. However, this high reliability comes at a cost of higher time consumption and costs incurred. To mitigate this, crowd or online tests could be used. Usually, online tests cover a wider range of end devices, environmental conditions, or participants, which may have an impact on the ratings. To verify whether such online tests can be used for visual quality assessment, we designed three online tests. These online tests are based on previously conducted lab tests as this enables comparison of the results of both test paradigms. Our focus is on the quality assessment of high-resolution images and videos. The online tests use AVrate Voyager, which is a publicly acces-sible framework for online tests. To transform the lab tests into online tests, dedicated adaptations in the test methodologies are required. The considered modifications are, for example, a patch-based or centre cropping of the images and videos, or a randomly sub-sampling of the to-be-rated stimuli. Based on the analysis of the test results in terms of correlation and SOS analysis it is shown that online tests can be used as a reliable replacement for lab tests albeit with some limitations. These limitations relate to, e.g., lack of appropriate display devices, limitation of web technologies, and modern browsers considering support for different video codecs and formats.