Investigation of the quality of audio encoding by different encoders


Free Download Mp4Gain
picture

Investigation of the quality of audio encoding by different encoders

sound quality

The MP3 format for high-quality audio encoding is becoming increasingly popular. Originally developed for use within the MPEG1 and MPEG2 video compression standards, it quickly became mainstream as a standalone format. The main reasons for this turn of events were the preservation of high sound quality with high compression ratios and the more than active attempts by the developer IIS Fraunhofer to make the most of his creation.

sound quality

In essence, MP3 is a direct evolution of MPEG Layer I and Layer II, and it also uses a psychoacoustic model to encode the original signal. Because of this, the encoding process is ambiguous and may vary depending on the encoder used (for more details, see MP3 Overview. Part 1). This ambiguity means that, having encoded the same signal with two different encoders, we can obtain, after decoding, two different sound signals. Obviously, the preferred encoder is the one that best preserves the original signal. The purpose of this review is to find out which modern encoder will give us the best result.

Test methodology
Generally, to compare the original and encoded signal, the method of comparing its amplitude frequency characteristics (AFC) is used. There are two varieties of this method: comparing the average frequency response of the signals, and comparing the change in frequency response over time. The first type is used most often due to the simplicity of its implementation: the comparison needs to be done only once. However, during averaging, a significant part of the information about the signal is lost, and as a consequence, with an absolutely identical frequency response, the original and encoded signal can differ greatly in sound if the signal contains a large amplitude. , but very short. bursts at term of some frequencies. The second type allows you to avoid such problems,

Not so long ago, a method for comparing sonograms of signals became widespread: a graphical representation of the frequency response of signals over time, in which time is plotted along the abscissa, the frequency of the component of the signal is plotted along the ordinate (generally a logarithmic scale is used), and the intensity of the luminescence of the points determines the amplitude of this frequency component of the signal. This method is essentially a modification of the method for comparing changes in frequency response over time, and the problem of underperformance is solved by reducing the number of frequency components considered and expanding the “scanning window” of the frequency response at 50-100. samples, allowing you to use FFT. This simplification of the method inevitably leads to a decrease in its precision. First, a decrease in the number of frequency components considered leads to a loss of the “resolution” of this method, making it practically analogous to averaging the frequency response over time. Second, due to the magnification of the scanned window and the use of the FFT, there is the effect of “smearing” the signal in time.


Free Download Mp4Gain
picture


Mp4Gain Main Window
picture


Mp4Gain Features
picture


Free Download Mp4Gain
picture

Sound via Bluetooth: versions and codecs, do they affect the sound quality?

Sound via Bluetooth: versions and codecs, do they affect the sound quality?

QUALITY AUDIO

Bluetooth is a wireless technology that allows, among other things, to connect headphones or portable speakers, stereos or sound bars to your smartphone or laptop.

Sound quality

There are many devices on the market today with different versions of the Bluetooth protocol and with support for different codecs. What does this all mean, which version is the latest, you need to understand all of this, and most importantly for us, does it affect the sound quality in any way? Let’s take a look.

One important thing to keep in mind up front is that the Bluetooth sound technology you hear has much less impact on sound quality than the design of the device itself. If you try to test multiple headphones or wireless speakers, you will hear the obvious differences between the devices. If you test the same device, but connected with different codecs or versions of Bluetooth, the difference may not be so obvious, maybe even subtle.

Simply put, you should not base your choice solely on the Bluetooth technologies and codecs supported by the device, this is neither the first nor the second most important criteria.

Bluetooth versions

Bluetooth has improved dramatically over the years since the turn of the century, and audio compression isn’t as brutal today as it was in the days of Bluetooth 1.1 or 2.0 (when wireless headphones and earphones first hit the market. ). Let us highlight the main technical differences between the Bluetooth versions, devices with which they can be found for sale today.

Bluetooth 3.0: The data transfer rate has increased significantly (up to 24 Mbps vs. 2.1 Mbps in Bluetooth 2.0), but the power consumption has increased.

Bluetooth 4.0: reduced power consumption while maintaining a speed of 24 Mbps.

Bluetooth 4.1: added anti-crosstalk protection when working with LTE modules installed on all 4G smartphones.

Bluetooth 4.2: higher speed and improved data transmission security.

Bluetooth 5.0: range increases 4 times, speed increases 2 times.

Bluetooth 5.1: it was possible to determine the location and direction with the utmost precision.

As you can see, the new versions primarily address data rate and power efficiency issues, which rarely directly affect sound quality. However, newer versions can improve data transfer reliability, which can allow a heavier codec, such as aptX HD or LDAC, to operate at higher data rates.

Remember that for a particular version of Bluetooth to work, both devices must support it. That is, if your smartphone has Bluetooth 5.1 and the headphones are 4.2, then Bluetooth 4.2 will be used for the connection.

Audio codecs and Bluetooth
So it is not the Bluetooth version that directly affects the final sound, but which audio codec is used for sound transmission. In fact, a codec here means a complex algorithm to process (encode / decode) audio data to send it wirelessly between your devices. Using a codec is necessary because the less data is transmitted over Bluetooth, the more reliable the connection is and the less likely your headphones are to lose signal in the middle of a favorite track.

All current Bluetooth codecs are “lossy”, discarding some of the audio data, reducing the audio data transfer rate (bit rate), for example CD quality with a speed of 1411 kilobits per second at about 300 ~ 900 kbps (roughly we’ll talk about Lossy and Lossless separately next time).

Another distinguishing feature of codecs is latency. The time it takes for the digital signal processor to decode the encoded audio. Long audio latency creates problems with your timing when watching a video.

As in the case of the BT version, for the codec to work, it is important that it be compatible not only with headphones or acoustics, but also with the sound source: a phone, a player or a computer.

SBC is standard for Bluetooth. This codec can operate at data rates of up to 345 kilobits per second. While the source and destination devices can handle high bit rate audio, the audio quality is difficult to distinguish from standard AAC or aptX quality. But with the lag it can exceed 100 milliseconds or more, which creates problems with audio timing when watching a video.

Sound quality

Sound quality

SOUND QUALITY

Although sound quality can be objectively measured with special instruments and devices, most people do not have such tools or knowledge to use them. To some extent, the quality can be judged subjectively, but it can hardly be called a reliable method. This article will help you decide what to focus on in terms of sound quality.

sound quality

MP3 is one of the most popular audio formats and a significant part of music is stored / distributed in this format. It is a lossy audio compression format, which means that the MP3 copy is slightly worse than the original. Some parts of the audio information are discarded when you convert audio to MP3. Like other modern lossy audio compression formats, it uses what is called “perceptual encoding.” This method mainly discards audio data that the human ear cannot pick up.

The main advantage of all lossy formats is the small file size (compared to lossless formats). Due to “perceptual coding”, this does not always mean that “lossy” files sound worse than “lossless” files. An MP3 file with a decent bit rate and frequency can only lose inaudible information. However, it should be noted that ear sensitivity is different for different people, so “inaudible” is a relative property.

An MP3 file has three main parameters: number of channels (stereo, mono, and joint stereo options), frequency (sampling), and bit rate. Stereo MP3 files of 44 kHz and 128 kbps are considered “sufficient” for music (at lower bit rates and frequencies, the quality decreases too much), but 192 kbps and more are recommended. Using bit rates higher than 320 Kbps is pointless as the size of such files is close to the size of “lossless” files, but compression is still lossy.

AudioConverter Studio – welcome

Lossless compression formats retain all the information in the original audio file, which means that they are identical copies of the audio. On the other hand, they are larger (sometimes significantly larger) than the same audio in lossy formats. Examples of lossless audio compression formats: FLAC, APE, WAV, MPC, etc.

For backups, lossless compression formats are the best option. Lossy formats are ideal for portable devices – a smaller file size allows you to store more songs in a limited space on your device. Although some portable players support lossless formats, the use of such formats on such devices is rarely justified.

Lossless formats are also the best option as an intermediate medium. For example, if you want to burn MIDI files as a music CD, you must first convert MIDI to one of the audio formats using MIDI Converter Studio, and then burn the resulting files using a program like Audio CD Burner Studio. You can of course convert MIDI to MP3 (lossy), but convert MIDI to WAV (lossless) more efficiently and then send the resulting WAV files to Audio CD Burner Studio.

It is possible to convert from a lossy format to a lossless format, but we will get large files with the same quality. Even though this kind of conversion doesn’t make sense, at least it won’t do any harm. As another example, we can consider converting MP3 from 128 Kbps to 320 Kbps. In this case, we not only get large files, but also worse quality. Remember that MP3 is a lossy audio compression format, that is, every time we convert to this format, some of the audio data is discarded.

How to measure sound or audio quality? Part 2

How to measure sound or audio quality? Part 2

Sound Quality

Epilogue to Steve Gutenberg’s Article

SOUND QUALITY

Since the article reflects a “controversial” view on measuring sound quality, we invited Yuri Fomin, speaker design engineer, founder of F-Lab, which develops acoustic systems for Defender, Jetbalance, AVE “and” Arslab “(the most famous of them:” Defender “G-2.1, Volcano 1;” Jetbalance “371, 381, 391;” AVE “DF 100, 102, 104;” Arslab “AC1, AC3):

“If you read the article carefully, there are two main aspects:

Thematic media often abuse the publication of graphic materials and technical characteristics that reflect the quality indicators of high-end loudspeaker systems and devices; often the publications themselves do not have a sufficient level of experience to comment on these indicators to make them understandable and useful to a lay person. As a result, these materials only further confuse consumers and cannot help promote the product.
When deciding to buy a particular system, you need to rely not only on the opinion of the experts, but also on your own feelings. You should not form an opinion on this or that high-end product based solely on the opinions of critics, bloggers, and online article authors. The fact is that equipment of this class is quite expensive. The purchase of such a system assumes that the user will not save on the acoustic design of the room (after all, it is this that allows him to hear all the delights of sound). Hence the conclusion: you should not rely on the experience of another person when listening to teams of advisers who (unlike the buyer) are not responsible for choosing their wallet and have listened to the operation of the products from time to time and outside the home ( especially if you consider that not all are professionals of this class,
So what kind of information is better to search on the Internet and in the media to choose high-quality equipment for yourself, if you have not yet formed an opinion on this matter?

When familiarizing yourself with the information published by the media, pay attention first to the data provided by the manufacturer or the trademark owner (they often form the basis of such publications about the product). If you see measurements or technical comments embedded in articles, remember that they are often done to create the “illusion of high competition.” The value can be provided by reports from manufacturers’ factories or historical references about brands, managers and engineers who were involved in the birth of the product.
Your attitude or friendliness can also make a significant contribution when choosing high-end products; After all, when developing, brands are not based on the opinions of the critics, but on their vision of the product and (most importantly) in a certain category of products. consumers, understanding their budgets, preferences and the conditions in which these people will use said products. So if you have formed a respectful attitude towards any brand, feel free to take the products of this manufacturer and you will not experience moral discomfort or disappointment.
However, if you decide to turn to blogs, reviews and articles from the Internet, remember that most of their authors are people, to put it mildly, who are not competent enough. The really useful information in this case may be the one that describes the situation of choosing a high-quality team like yours. “

How to measure sound or audio quality?

How to measure sound or audio quality?

sound quality

I’ve met quite a few audio engineers and sound engineers in my life, and the best of them had one thing in common: they had great ears.

sound quality

They knew what a good sound was. The field of their opponents is occupied by engineers who rely solely on measurements to “prove” that “their” sound is better. In my opinion, people in the second group rarely create good-sounding products. Audio is too complex to be analyzed by numbers alone.

Today I see more and more digital audio engineers specializing in designing programs to correct for speaker characteristics and room acoustics. They are generally very nice people, and their diagrams and tests look pretty impressive when displayed from a laptop, but this all fades into the background, once I hear real sound. Results often range from good to really scary – your hard work is the least likely to succeed.

Obviously, they were so focused on measuring the characteristics of the sound that they forgot to listen to the result, or at least hire assistants who understood what good sound is. If the goal of these people was to achieve the best possible performance, I would congratulate them on their successful implementation. But they strive for something different, and understanding exactly what kinds of measures characterize improvement in sound quality is an art. Art in which only a few have succeeded.

There are several reasons why the measurements do not correlate with subjective evaluations of sound quality. First, measuring the characteristics of electronics and speakers has little to do with how the music sounds. The test tones are too simple and predictable; the music is much more complex and varied. Reproducing the sound of a violin or drums is extremely challenging, and since the real purpose of any hi-fi system is to play music, not test tones, the most important task for an engineer should be to develop a product that sounds “good.” . to the target audience. For example, if he is designing DJ headphones, he is not trying to reproduce the bass as accurately as possible. Amplifier designers shouldn’t waste time trying to design an amplifier that delivers maximum performance during test loads. They must create amplifiers that can handle the complexities of playing music with real speakers. Amp designers don’t know which speakers their product will ultimately work with. Each type of speaker places a different “load” on the amplifier.

In the 1970s, a new generation of amplifiers emerged that had much lower “harmonic distortion” than their predecessors; It was believed that if this value could be measured more accurately, the amplifiers would produce a better sound. But they did not measure what was really needed. Blindfolded perception, expressed in “bespoke” design, is unlikely to take sound reproduction to a qualitatively new level. This does not mean that metric values ​​are useless; They are a great help to talented engineers, but measurements cannot replace listening to music.

I know the sound of the latest audio adjustment and calibration systems installed in receivers from Denon, Onkyo, Pioneer, Sony and Yamaha, and I can say that the result of their work is difficult to predict in advance. Of course, these systems can improve the sound, but most of the time, processing the sound just changes it. In these cases, I get the best results by listening to music and manually tuning in parallel. Processed audio may have better numerical characteristics, but again, this does not necessarily correlate with improved audio quality.

I recently talked about this with a friend of mine, writer Brent Butterworth, who believes that quantification is a useful tool, but we never reached a compromise. However, according to him, the measurements of the value of the indicators that characterize flaws in the speaker’s sound may go unnoticed by the human ear, which means that some speakers, characterized by not having the most optimal performance, from a point of view subjective, they can produce high quality sound. So find out.

Therefore, even if experienced engineers are faced with the problem of separating useful information from unnecessary quantitative indicators, it is difficult to imagine how a buyer looking at the wavy lines on a graph can use it to decide which amplifier or speakers to buy.

SOUND QUALITY

SOUND QUALITY

Sound Quality

A product with high quality sound is well received by customers, sells successfully and increases brand equity.

sound quality

From automobiles to hand tools to heavy machinery, consumers believe that sound is a key indicator of the overall quality level of any product. The noise inevitably comes from the products, but it can often be given a proper sound, for example the massive thud of the door closing on an expensive car. Sometimes the sound is even desirable; For example, a motorcycle with a powerful and vibrant engine has a chance to please the buyer. A product with high quality sound is well received by customers, sells successfully, and increases brand equity. More than sound levels Designing a product with sound quality in mind is challenging because the “desired sound quality” “is determined by how we perceive it with our ears and brain. For example, measuring the frequency response of sound describes its timbre, but not the physical and cognitive processes that occur in our ears and brain. However, these processes affect our subjective perception of sound. Sound quality needs to be quantified to provide designers with clear goals for the big picture and to ensure sound engineering effectiveness Objective and subjective sound quality Since some people perceive sound more easily than others, a factor must be included weighting as the Zwicker volume in sound quality metrics. Only the human ear can tell if a product sounds good and therefore subjective testing of customer expectations plays an important role in determining acceptable sound quality. In group tests, listeners rate sounds and quantify quality metrics such as harshness. Analytical software converts these estimates into target values ​​that engineers can use as targets (for example, frequency response). This methodology can be used during product development to evaluate design changes.

The concepts of pixel, video pixel, point.

The concepts of pixel, video pixel, point.

Pixel

In computer graphics, the term “pixel”, broadly speaking, can mean different concepts: the smallest element of the image on a computer screen; a separate bitmap element; dot of the image printed on the printer.

pixel

Therefore, to avoid confusion, use the following terminology: in ideopixel: the smallest element of the image on the screen; pixel – a separate element of the raster image; dot is the smallest item produced by the printer.

The display screen is divided into a fixed number of video pixels, which form a graphical grid (raster) of a fixed number of rows and columns. The size of the graphics grid is generally expressed in the form NxM, where N is the horizontal number of video pixels and M is the vertical number. On modern displays, graphical grids are used: 640 x 480, 800 x 600, 1024 x 768, 1240 x 1024, etc. The image on the display screen is created by selectively illuminating certain video pixels on the screen with an electron beam. For an image to be perceived by the eye, it must be made up of hundreds or thousands of video pixels, each of which must be illuminated. ”

Raster graphics are photo quality images. If the pixel sizes are small enough (close to the size of video pixels), then the raster image looks no worse than a photograph. Raster graphics effectively represent photo-quality images.

A large amount of memory is required to store bitmaps.

The bitmap file stores the color information for each video pixel as a combination of bits. Simple bitmap images take up a small amount of memory (several tens or hundreds of kilobytes). Photo-quality images typically require several megabytes. Therefore, a large amount of memory is required to store bitmaps.

Bitmap becomes distorted when scaling.

The bitmap may lose its appeal after scaling or rotating it. For example, areas of solid color can take on an odd pattern (“moire”); curves and straight lines that seemed smooth can suddenly become sawtooth. If you reduce and then enlarge the bitmap to its previous size, it will become blurry and uneven, and the filled areas may be distorted. Bitmaps have very limited options for scaling, rotating, and other transformations

Tasks for which raster programs are used.

Raster graphics allow you to create photo-quality images. After all, what are raster graphics? It is an image that consists of pixels, small squares, each of which is assigned a specific color. On any screen, the information is displayed exactly in raster form.

What do the concepts of pixel, video pixel, point mean?

In computer graphics, the term “pixel”, in general terms, can mean different concepts:

· The smallest element of the image on the computer screen;

· A separate element of the bitmap image;

· Dot of the image printed on the printer.

Therefore, to avoid confusion, we will use the following terminology:

· Video pixel: the smallest picture element on the screen;

· Pixel: a separate element from the raster image;

· Dot: the smallest element produced by the printer.

In this case, for the one-pixel image on the computer screen, one or more video pixels can be used.

WHAT ARE VIDEO RESOLUTIONS? Part 2

WHAT ARE VIDEO RESOLUTIONS? Part 2

Video Resolution

Choosing HD CCTV Camera

Video Quality

Besides image resolution, what else should you consider when choosing HD network cameras? Here we will share information on how to choose the right HD cameras from the installer’s point of view.

Poor light
As you know, a CCTV camera works differently from a home camera – a CCTV camera cannot use a flash when capturing an image / video. If the camera performs poorly in low light, its use is limited. When working in low light conditions, said camera “ fades ”, despite its very high resolution.

High resolution is a double-edged sword: the sensor manufacturer does not have the ability to infinitely increase the area of ​​the matrix, therefore the increase in resolution is associated with a decrease in the size of the pixel itself with the Same size of the sensor array (usually 1/3 ”), so there is less light for each pixel, leading to decreased sensitivity with increasing resolution (megapixels).

Currently, the optimal value for most areas of video surveillance is the resolution of 2MP (1080p / FullHD), it is for this resolution that most of the sensors of the Low Illumination series exist.

Time delay
All network (IP) CCTV cameras have some latency compared to real time, and the cost or quality of the camera is not a determining factor in this latency. For example, for the same 720p image, the video delay time for some cameras is 0.1 s, and for some other network cameras this time can be 0.4 s, or even more than 0.7 s. Why is the video delay time different? Unlike an analog camera, a network camera compresses video (a process called encoding) and user devices decode the video for display, resulting in video lag. Generally, the lower the latency, the better the image processor’s capabilities. This means choosing the network camera with the lowest video latency.

Heat dissipation
When the CCTV camera is in operation, it generates heat, especially when the infrared light is turned on at night. This rule is valid for any CCTV camera. Excessive heat generation increases the likelihood of overheating and consequently damage to the camera. When choosing megapixel cameras, pay attention to:

Choose a camera with lower power consumption. Low power consumption means that the camera saves energy and generates less heat. The downside: in winter, a camera with low heat emission can freeze (it is usually about the infrared filter), and also low consumption means that weak infrared illumination is installed, this must also be taken into account.

Consider using a camera with improved performance in low light conditions (no infrared or other artificial lighting). Such a camera in low light conditions can capture images even in the dark (> 0.009 – 0.001 lux).

Choose a camera with good heat dissipation. A metal box is preferable to a plastic one. To ensure reliable performance, the Elite series network cameras use a finned heat sink in the housing to maximize heat dissipation, greatly helping the camera to ensure reliable performance.

Price
“High price = high quality” – In most cases, this rule is true. Based on research reports, it can be said that the consumer often believes that a higher price of a product indicates a higher level of quality. But price is not the only indicator of good quality, especially when buying Made in China products. I have been in the video surveillance industry for more than five years and I can affirm that end users, integrators and installers can get high quality products from Chinese suppliers / manufacturers at a very competitive price. High-end cameras can have unique body designs, offering special features not found in other products.

Technical support
In conclusion, I would like to say that network cameras should also have good technical support. Although IP cameras are becoming easier to set up and operate, end users may encounter technical problems that require third party assistance. Faced with such problem, you will receive our technical support within 1-2 days, this is quite acceptable. This is why I personally do not recommend buying CCTV cameras on Aliexpress, as in the future you are unlikely to receive technical support from online sellers.

WHAT ARE VIDEO RESOLUTIONS?

WHAT ARE VIDEO RESOLUTIONS?

video resolution

1. What is a high definition CCTV camera?

Video Resolution

All image formats with a resolution of 1280×720 are considered high definition (HD) format. In the modern world of video surveillance, there are two areas: analog and digital. Consequently, there are analog and network (IP) HD cameras. 960H (NTSC: 960×480) resolution is not HD. Current HD resolution formats include: 1.0 megapixels (720p), 1.3 megapixels (960p), 2 megapixels (1080p), 3 megapixels, 5 megapixels, 8 megapixels (4K UHD), 12 megapixels, 33 megapixels (8K UHD) .. .
Typically, HD network cameras provide slightly better image quality than analog HD cameras of the same resolution (eg 720p).
Recently one of our customers reported that they installed a video surveillance system on 720p AHD cameras (the manufacturer said 1000TVL) and was not satisfied – the image quality of these 720p AHD cameras was even worse than that of the old 960H cameras. Why this happened, we will tell in the fourth part of the article.

2. Advantages of high definition
Compared to standard definition, HD technology has increased image detail. Image quality is further enhanced by various enhancement technologies such as progressive scan, 2D / 3D dynamic noise reduction, wide dynamic range (WDR), etc. In short, HD offers excellent image quality. A typical 960H analog camera offers a resolution of 960H / WD1, which is 960×480 pixels (for NTSC) or 960×576 pixels (for PAL). Once the signal is digitized in the DVR or HVR, the image will be a maximum of 552960 pixels (0.5 megapixels).
A high definition camera can cover a much larger area than a conventional camera. Take, for example, a 12MP fisheye lens panoramic camera with a 360-degree field of view. With an integrated 12 megapixel image sensor and ePTZ (virtual pan / tilt / zoom) and split image capabilities, it can replace multiple conventional CCTV cameras, significantly reducing installation costs and subsequent maintenance fees.
Excellent compatibility is another advantage of HD. Whether you shop online or go to your local electronics store, you will notice that all televisions, camcorders, and digital cameras support HD 1080p (FullHD). Consequently, if you want this equipment to work with your CCTV system, you should choose a CCTV system that supports 1080p. We also understand that 4K is the current trend, it is logical to expect 4K UHD video surveillance system to become popular in the future.

5MP resolution

3. Various HD resolution formats
High definition IP cameras are at the forefront of video surveillance systems. They can provide higher quality video with greater image detail and wider coverage than standard definition cameras. You can select the desired format of network (IP) cameras according to your requirements. For example, for license plate or facial recognition applications, choose megapixel network cameras with a resolution of 1080p or higher. For the resolution of a particular HD format, see the following table:

Format Resolution (pixels) Aspect Ratio Scan
1MP / 720P 1280 x 720 16: 9 Progressive
SXGA / 960P 1280 x 960 4: 3 Progressive
1.3 MP 1280×1024 5: 4 Progressive
2MP / 1080P 1920 x 1080 16: 9 Progressive
2.3 megapixels 1920×1200 16:10 Progressive
3MP 2048×1536 4: 3 Progressive
4MP 2592×1520 16: 9 Progressive
5MP 2560×1960 4: 3 Progressive
6MP 3072×2048 3: 2 Progressive
4K Ultra HD 3840×2160 16: 9 Progressive
8K Ultra HD 7680×4320 16: 9 Progressive

Video quality evaluation. Subjective evaluation of video quality Part 2

Video quality evaluation. Subjective evaluation of video quality Part 2

Video Quality

Although subjective testing was carried out many times by various organizations, until recently there were no stable testing programs designed to work with personal computers in the public domain.

video quality

This was the reason for the development of MSU’s perceptual video quality tool, which implements various methods for subjective comparison and analysis of results.

The subjective test method is a combination of methods to demonstrate sequences, gather expert opinions, and process the results.
Let us consider, using an example of video codec comparison, the test procedure by the SAMVIQ method, recently developed at the EBU (European Broadcasting Union), the implementation of the MSU Perceptual Video Quality tool. This method was used in the subjective comparison of modern video codecs.

SAMVIQ method diagram
Test stages:

1. Expert enters her name (any unique sequence of characters).

2. Color perception test (standard Ishihara charts are used).

3. For each test sequence:

The reference video (original) is shown.
Whenever there are unseen compressed versions of this video, the expert selects the next version of the video, watches it, and rates it. The movie rating belongs to the 0-100 segment, the higher the better. The evaluation of the sequence variants already reviewed can be changed at any time, it is also possible to review any of the variants.
If all variants of the video have been viewed, the expert can proceed to the next test sequence.
Different variants of the compressed sequence are hidden behind the letter designations, so the expert does not know which codec he is evaluating at the moment. The reference video is explicitly available, it is also hidden under one of the letter designations and is rated on a par with the compressed video streams.

Why are such complications necessary? There are several problems that subjective testing techniques should solve. The first of these is to create a common rating scale for all experts, that is, so that the “good” rating means roughly the same for different experts. This is achieved through a technique called “anchoring”: during the test, both the video with the highest quality (“high anchor”, must be associated by all the experts with the maximum score), and with the lowest (“low anchor “, should be associated with a minimum score).

Another task is to minimize the memory effect, the influence of the sequence in which the video is shown on the expert evaluations. In some test methods, this problem is solved by displaying the reference video (original) together with each processed video sequence. In the SAMVIQ method, which we used in the comparison, the first problem is solved using a hidden and explicitly available reference video, and the second, using a more flexible evaluation procedure than in other methods (an expert can review the video and change your evaluations).

With any test method, subjective test results can be influenced by many external factors. It is essential that all testers are instructed on how to pass the test, that there is adequate lighting in the room, and that the tests should not tire the experts. Anything from the gender of the experts to their professions to the timing of testing can make a difference in results. Interestingly, compared to all other factors, monitor characteristics (resolution, LCD / CRT, etc.) do not significantly affect results (see M. Pinson, S. Wolf, “The impact of monitor type and resolution in the subjective video Quality tests “NTIA TM-04-412). Processing of results

The main results are obtained after a simple average of the evaluations by experts. The resulting score is called the MOS (Mean Opinion Score). In addition, to evaluate the dissemination of opinions, a confidence interval is usually given (the interval in which the real mean opinion is located with a certain probability). There are techniques to exclude experts that give results that are unstable and very different from the average.

At the end of 2005, our laboratory carried out subjective tests of video codecs. The test tasks were the subjective comparison of new versions of popular codecs, the comparison of results with objective metric data, and the development of subjective test technology. This article contains only a part of the results obtained.