
Video quality evaluation. Subjective evaluation of video quality

What is quality?

Dahl’s dictionary contains the following definition: “quality is a property or belonging, everything that constitutes the essence of a person or thing”. What if we need to evaluate the visual quality, for example, of a video compressed with a codec? You can understand the quality of a video as an average rating of the opinions of the people who watch that video. It is this indicator that the creators of video processing systems ultimately want to improve, so I would like to be able to estimate it numerically. There are two approaches to solving this problem: subjective and objective evaluation of video quality. In the first part of this article, we will talk about these approaches, and in the second, we will present the results of our subjective comparison of modern video codecs.
Objective test
You can evaluate video quality using a formula or algorithm such as PSNR, VQM, or SSIM (see Comparing Video Codecs to PSNR Metrics). The main advantage of this approach is the ability to automate the testing process, allowing you to measure the rendering quality of a video system with a large number of different settings and test videos, or measure the quality in real time. The measurements also provide accurate and reproducible data. The downside to this approach is that automated metrics can misrepresent the subjectively perceived quality. This can lead, for example, to an erroneous conclusion about the superiority of one codec over another.
An alternative way to get an estimate of the video quality is to perform subjective tests. The idea behind this method is to get the quality score directly from the viewers who are evaluating the video. A similar approach has long been used to evaluate sound quality. For example, the hydrogenaudio.org forum regularly hosts subjective tests of audio codecs. What is needed to perform subjective tests?
Select video footage to test. Typically, a video of about 8-10 seconds is used to prevent experts from dispersing attention and shortening the overall time of the experiment.
Select the settings for the video processing systems you want to compare.
Choose a test method.
Invite a sufficient number of experts (at least 15 recommended).
Based on your opinion, get final grades.
In 1974 the first version of ITU-R BT.500 “Methodology for the subjective assessment of the quality of television pictures” was published. These guidelines provide a complete description of how all of the above issues should be addressed. Since then, many subjective tests have been carried out, the most recent of which is the Subjective Quality Assessment of the Emerging AVC / H.264 Coding Standard, Subjective Quality of Internet Video Codecs. Volumetric subjective tests are performed by VQEG (Video Quality Experts Group).













nd more than 450 thousand conversions on an OS X operating system.

