One document matched: draft-ietf-netvc-testing-02.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="rfc2629.xslt" ?>
<!-- generated by https://github.com/cabo/kramdown-rfc2629 version 1.0.20 -->
<!DOCTYPE rfc SYSTEM "rfc2629.dtd" [
]>
<rfc ipr="trust200902" docName="draft-ietf-netvc-testing-02" category="info">
<?rfc toc="yes"?>
<?rfc sortrefs="yes"?>
<?rfc symrefs="yes"?>
<front>
<title>Video Codec Testing and Quality Measurement</title>
<author initials="T." surname="Daede" fullname="Thomas Daede">
<organization>Mozilla</organization>
<address>
<email>tdaede@mozilla.com</email>
</address>
</author>
<author initials="A." surname="Norkin" fullname="Andrey Norkin">
<organization>Netflix</organization>
<address>
<email>anorkin@netflix.com</email>
</address>
</author>
<date year="2016" month="March" day="15"/>
<area>RAI</area>
<keyword>Internet-Draft</keyword>
<abstract>
<t>This document describes guidelines and procedures for evaluating a video codec specified at the IETF. This covers subjective and objective tests, test conditions, and materials used for the test.</t>
</abstract>
</front>
<middle>
<section anchor="introduction" title="Introduction">
<t>When developing a video codec, changes and additions to the codec need to be decided based on their performance tradeoffs. In addition, measurements are needed to determine when the codec has met its performance goals. This document specifies how the tests are to be carried about to ensure valid comparisons when evaluating changes under consideration. Authors of features or changes should provide the results of the appropriate test when proposing codec modifications.</t>
</section>
<section anchor="subjective-quality-tests" title="Subjective quality tests">
<t>Subjective testing is the preferable method of testing video codecs.</t>
<t>Because the IETF does not have testing resources of its own, it has to rely on the resources of its participants. For this reason, even if the group agrees that a particular test is important, if no one volunteers to do it, or if volunteers do not complete it in a timely fashion, then that test should be discarded. This ensures that only important tests be done in particular, the tests that are important to participants.</t>
<section anchor="still-image-pair-comparison" title="Still Image Pair Comparison">
<t>A simple way to determine superiority of one compressed image over another is to visually compare two compressed images, and have the viewer judge which one has a higher quality. This is mainly used for rapid comparisons during development. For this test, the two compressed images should have similar compressed file sizes, with one image being no more than 5% larger than the other. In addition, at least 5 different images should be compared.</t>
</section>
<section anchor="subjective-viewing-test" title="Subjective viewing test">
<t>A subjective viewing test is the preferred method of evaluating the quality. The subjective test should be performed as either consecutively showing the video sequences on one screen or on two screens located side-by-side. The testing procedure should normally follow rules described in <xref target="BT500"/> and be performed with non-expert test subjects. The result of the test could be (depending on the test procedure) mean opinion scores (MOS) or differential mean opinion scores (DMOS). Normally, confidence intervals are also calculated to judge whether the difference between two encodings is statistically significant.</t>
</section>
<section anchor="expert-viewing" title="Expert viewing">
<t>An expert viewing test can be performed in the case when an answer to a particular question should be found. An example of such test can be a test involving video coding experts on evaluation of a particular problem, for example such as comparing the results of two de-ringing filters. Depending on what information is sought, the appropriate test procedure can be chosen.</t>
</section>
</section>
<section anchor="objective-metrics" title="Objective Metrics">
<t>Objective metrics are used in place of subjective metrics for easy and repeatable experiments. Most objective metrics have been designed to correlate with subjective scores.</t>
<t>The following descriptions give an overview of the operation of each of the metrics. Because implementation details can sometimes vary, the exact implementation is specified in C in the Daala tools repository <xref target="DAALA-GIT"/>.</t>
<t>Unless otherwise specified, all of the metrics described below only apply to the luma plane, individually by frame. When applied to the video, the scores of each frame are averaged to create the final score.</t>
<t>Codecs are allowed to internally use downsampling, but must include a normative upsampler, so that the metrics run at the same resolution as the source video. In addition, some metrics, such as PSNR and FASTSSIM, have poor behavior on downsampled images, so it must be noted in test results if downsampling is in effect.</t>
<section anchor="overall-psnr" title="Overall PSNR">
<t>PSNR is a traditional signal quality metric, measured in decibels. It is directly drived from mean square error (MSE), or its square root (RMSE). The formula used is:</t>
<t>20 * log10 ( MAX / RMSE )</t>
<t>or, equivalently:</t>
<t>10 * log10 ( MAX^2 / MSE )</t>
<t>where the error is computed over all the pixels in the video, which is the method used in the dump_psnr.c reference implementation.</t>
<t>This metric may be applied to both the luma and chroma planes, with all planes reported separately.</t>
</section>
<section anchor="frame-averaged-psnr" title="Frame-averaged PSNR">
<t>PSNR can also be calculated per-frame, and then the values averaged together. This is reported in the same way as overall PSNR.</t>
</section>
<section anchor="psnr-hvs-m" title="PSNR-HVS-M">
<t>The PSNR-HVS metric performs a DCT transform of 8x8 blocks of the image, weights the coefficients, and then calculates the PSNR of those coefficients. Several different sets of weights have been considered. <xref target="PSNRHVS"/> The weights used by the dump_pnsrhvs.c tool in the Daala repository have been found to be the best match to real MOS scores.</t>
</section>
<section anchor="ssim" title="SSIM">
<t>SSIM (Structural Similarity Image Metric) is a still image quality metric introduced in 2004 <xref target="SSIM"/>. It computes a score for each individual pixel, using a window of neighboring pixels. These scores can then be averaged to produce a global score for the entire image. The original paper produces scores ranging between 0 and 1.</t>
<t>For the metric to appear more linear on BD-rate curves, the score is converted into a nonlinear decibel scale:</t>
<t>-10 * log10 (1 - SSIM)</t>
</section>
<section anchor="multi-scale-ssim" title="Multi-Scale SSIM">
<t>Multi-Scale SSIM is SSIM extended to multiple window sizes <xref target="MSSSIM"/>.</t>
</section>
<section anchor="fast-multi-scale-ssim" title="Fast Multi-Scale SSIM">
<t>Fast MS-SSIM is a modified implementation of MS-SSIM which operates on a limited number of scales and with modified weights <xref target="FASTSSIM"/>. The final score is converted to decibels in the same manner as SSIM.</t>
</section>
<section anchor="ciede2000" title="CIEDE2000">
<t>CIEDE2000 is a metric based on CIEDE color distances <xref target="CIEDE2000"/>. It generates a single score taking into account all three chroma planes. It does not take into consideration any structural similarity or other psychovisual effects.</t>
</section>
<section anchor="vmaf" title="VMAF">
<t>Video Multi-method Assessment Fusion (VMAF) is a full-reference perceptual video quality metric that aims to approximate human perception of video quality <xref target="VMAF"/>. This metric is focused on quality degradation due compression and rescaling. VMAF estimates the perceived quality score by computing scores from multiple quality assessment algorithms, and fusing them using a support vector machine (SVM). Currently, three image fidelity metrics and one temporal signal have been chosen as features to the SVM, namely Anti-noise SNR (ANSNR), Detail Loss Measure (DLM), Visual Information Fidelity (VIF), and the mean co-located pixel difference of a frame with respect to the previous frame.</t>
</section>
</section>
<section anchor="comparing-and-interpreting-results" title="Comparing and Interpreting Results">
<section anchor="graphing" title="Graphing">
<t>When displayed on a graph, bitrate is shown on the X axis, and the quality metric is on the Y axis. For publication, the X axis should be linear. The Y axis metric should be plotted in decibels. If the quality metric does not natively report quality in decibels, it should be converted as described in the previous section.</t>
</section>
<section anchor="bjontegaard" title="Bjontegaard">
<t>The Bjontegaard rate difference, also known as BD-rate, allows the measurement of the bitrate reduction offered by a codec or codec feature, while maintaining the same quality as measured by objective metrics. The rate change is computed as the average percent difference in rate over a range of qualities. Metric score ranges are not static - they are calculated either from a range of bitrates of the reference codec, or from quantizers of a third, reference codec. Given a reference codec, test codec, and ranges, BD-rate values are calculated as follows:</t>
<t><list style="symbols">
<t>Rate/distortion points are calculated for the reference and test codec. There need to be enough points so that at least four points lie within the quality levels.</t>
<t>The rates are converted into log-rates.</t>
<t>A piecewise cubic hermite interpolating polynomial is fit to the points for each codec to produce functions of distortion in terms of log-rate.</t>
<t>Metric score ranges are computed.
<list style="symbols">
<t>If using a bitrate range, metric score ranges are computed by converting the rate bounds into log-rate and then looking up scores of the reference codec using the interpolating polynomial.</t>
<t>If using a quantizer range, a third anchor codec is used to generate metric scores for the quantizer bounds. The anchor codec makes the range immune to quantizer changes.</t>
</list></t>
<t>The log-rate is numerically integrated over the metric range for each curve.</t>
<t>The resulting integrated log-rates are converted back into linear rate, and then the percent difference is calculated from the reference to the test codec.</t>
</list></t>
</section>
<section anchor="ranges" title="Ranges">
<t>For all tests described in this document, quantizers of an anchor codec are used to determine the quality ranges. The anchor codec used for ranges is libvpx 1.5.0 run with VP9 and High Latency CQP settings. The quality range used is that achieved between cq-level 20 and 60.</t>
</section>
</section>
<section anchor="test-sequences" title="Test Sequences">
<section anchor="sources" title="Sources">
<t>Lossless test clips are preferred for most tests, because the structure of compression artifacts in already-compressed clips may introduce extra noise in the test results. However, a large amount of content on the internet needs to be recompressed at least once, so some sources of this nature are useful. The encoder should run at the same bit depth as the original source. In addition, metrics need to support operation at high bit depth. If one or more codecs in a comparison do not support high bit depth, sources need to be converted once before entering the encoder.</t>
</section>
<section anchor="test-sets" title="Test Sets">
<t>Sources are divided into several categories to test different scenarios the codec will be required to operate in. For easier comparison, all videos in each set should have the same color subsampling, same resolution, and same number of frames. In addition, all test videos must be publicly available for testing use, to allow for reproducibility of results. All current test sets are available for download <xref target="TESTSEQUENCES"/>.</t>
<t><list style="symbols">
<t>Still images are useful when comparing intra coding performance. Xiph.org has four sets of lossless, one megapixel images that have been converted into YUV 4:2:0 format. There are four sets that can be used:
<list style="symbols">
<t>subset1 (50 images)</t>
<t>subset2 (50 images)</t>
<t>subset3 (1000 images)</t>
<t>subset4 (1000 images)</t>
</list></t>
<t>video-hd-3, a set that consists of 1920x1080 clips from <xref target="DERFVIDEO"/> (1500 frames total)</t>
<t>vc-360p-1, a low quality video conferencing set (2700 frames total)</t>
<t>vc-720p-1, a high quality video conferencing set (2750 frames total)</t>
<t>netflix-4k-1, a cinematic 4K video test set (2280 frames total)</t>
<t>netflix-2k-1, a 2K scaled version of netflix-4k-1 (2280 frames total)</t>
<t>twitch-1, a game sequence set (2280 frames total)</t>
</list></t>
</section>
<section anchor="operating-points" title="Operating Points">
<t>Four operating modes are defined. High latency is intended for on demand streaming, one-to-many live streaming, and stored video. Low latency is intended for videoconferencing and remote access. Both of these modes come in CQP and unconstrained variants. When testing still image sets, such as subset1, high latency CQP mode should be used.</t>
<section anchor="common-settings" title="Common settings">
<t>Encoders should be configured to their best settings when being compared against each other:</t>
<t><list style="symbols">
<t>vp10: –codec=vp10 –ivf –frame-parallel=0 –tile-columns=0 –cpu-used=0 –threads=1</t>
</list></t>
</section>
<section anchor="high-latency-cqp" title="High Latency CQP">
<t>High Latency CQP is used for evaluating incremental changes to a codec. It should not be used to compare unrelated codecs to each other. It allows codec features with intrinsic frame delay.</t>
<t><list style="symbols">
<t>daala: -v=x -b 2</t>
<t>vp9: –end-usage=q –cq-level=x -lag-in-frames=25 -auto-alt-ref=2</t>
<t>vp10: –end-usage=q –cq-level=x -lag-in-frames=25 -auto-alt-ref=2</t>
</list></t>
</section>
<section anchor="low-latency-cqp" title="Low Latency CQP">
<t>Low Latency CQP is used for evaluating incremental changes to a codec. It should not be used to compare unrelated codecs to each other. It requires the codec to be set for zero intrinsic frame delay.</t>
<t><list style="symbols">
<t>daala: -v=x</t>
<t>vp10: –end-usage=q –cq-level=x -lag-in-frames=0</t>
</list></t>
</section>
<section anchor="unconstrained-high-latency" title="Unconstrained High Latency">
<t>The encoder should be run at the best quality mode available, using the mode that will provide the best quality per bitrate (VBR or constant quality mode). Lookahead and/or two-pass are allowed, if supported. One parameter is provided to adjust bitrate, but the units are arbitrary. Example configurations follow:</t>
<t><list style="symbols">
<t>x264: –crf=x</t>
<t>x265: –crf=x</t>
<t>daala: -v=x -b 2</t>
<t>vp10: –end-usage=q –cq-level=x -lag-in-frames=25 -auto-alt-ref=2</t>
</list></t>
</section>
<section anchor="unconstrained-low-latency" title="Unconstrained Low Latency">
<t>The encoder should be run at the best quality mode available, using the mode that will provide the best quality per bitrate (VBR or constant quality mode), but no frame delay, buffering, or lookahead is allowed. One parameter is provided to adjust bitrate, but the units are arbitrary. Example configurations follow:</t>
<t><list style="symbols">
<t>x264: –crf-x –tune zerolatency</t>
<t>x265: –crf=x –tune zerolatency</t>
<t>daala: -v=x</t>
<t>vp10: –end-usage=q –cq-level=x -lag-in-frames=0</t>
</list></t>
</section>
</section>
</section>
<section anchor="automation" title="Automation">
<t>Frequent objective comparisons are extremely beneficial while developing a new codec. Several tools exist in order to automate the process of objective comparisons. The Compare-Codecs tool allows BD-rate curves to be generated for a wide variety of codecs <xref target="COMPARECODECS"/>. The Daala source repository contains a set of scripts that can be used to automate the various metrics used. In addition, these scripts can be run automatically utilizing distributed computers for fast results, with the AreWeCompressedYet tool <xref target="AWCY"/>. Because of computational constraints, several levels of testing are specified.</t>
<section anchor="regression-tests" title="Regression tests">
<t>Regression tests run on a small number of short sequences. The regression tests should include a number of various test conditions. The purpose of regression tests is to ensure bug fixes (and similar patches) do not negatively affect the performance. The anchor in regression tests is the previous revision of the codec in source control. Regression tests are run on the following sets, in both high and low latency CQP modes:</t>
<t><list style="symbols">
<t>vc-720p-1</t>
<t>netflix-2k-1</t>
</list></t>
</section>
<section anchor="objective-performance-tests" title="Objective performance tests">
<t>Changes that are expected to affect the quality of encode or bitstream should run an objective performance test. The performance tests should be run on a wider number of sequences. If the option for the objective performance test is chosen, wide range and full length simulations are run on the site and the results (including all the objective metrics) are generated. Objective performance tests are run on the following sets, in both high and low latency CQP modes:</t>
<t><list style="symbols">
<t>video-hd-3</t>
<t>netflix-2k-1</t>
<t>netflix-4k-1</t>
<t>vc-720p-1</t>
<t>vc-360p-1</t>
<t>twitch-1</t>
</list></t>
</section>
<section anchor="periodic-tests" title="Periodic tests">
<t>Periodic tests are run on a wide range of bitrates in order to gauge progress over time, as well as detect potential regressions missed by other tests.</t>
</section>
</section>
</middle>
<back>
<references title='Informative References'>
<reference anchor="PSNRHVS" >
<front>
<title>A New Full-Reference Quality Metrics Based on HVS</title>
<author initials="K." surname="Egiazarian" fullname="Karen Egiazarian">
<organization></organization>
</author>
<author initials="J." surname="Astola" fullname="Jaakko Astola">
<organization></organization>
</author>
<author initials="N." surname="Ponomarenko" fullname="Nikolay Ponomarenko">
<organization></organization>
</author>
<author initials="V." surname="Lukin" fullname="Vladimir Lukin">
<organization></organization>
</author>
<author initials="F." surname="Battisti" fullname="Federica Battisti">
<organization></organization>
</author>
<author initials="M." surname="Carli" fullname="Marco Carli">
<organization></organization>
</author>
<date year="2002"/>
</front>
</reference>
<reference anchor="FASTSSIM" target="http://live.ece.utexas.edu/publications/2011/chen_rtip_2011.pdf">
<front>
<title>Fast structural similarity index algorithm</title>
<author initials="M." surname="Chen" fullname="Ming-Jun Chen">
<organization></organization>
</author>
<author initials="A.C." surname="Bovik" fullname="Alan Conrad Bovik">
<organization></organization>
</author>
<date year="2010"/>
</front>
</reference>
<reference anchor="MSSSIM" target="http://www.cns.nyu.edu/~zwang/files/papers/msssim.pdf">
<front>
<title>Multi-Scale Structural Similarity for Image Quality Assessment</title>
<author initials="Z." surname="Wang" fullname="Zhou Wang">
<organization></organization>
</author>
<author initials="E.P." surname="Simoncelli" fullname="Eero P. Simoncelli">
<organization></organization>
</author>
<author initials="A.C." surname="Bovik" fullname="Alan Conrad Bovik">
<organization></organization>
</author>
<date year="n.d."/>
</front>
</reference>
<reference anchor="SSIM" target="http://www.cns.nyu.edu/pub/eero/wang03-reprint.pdf">
<front>
<title>Image Quality Assessment: From Error Visibility to Structural Similarity</title>
<author initials="Z." surname="Wang" fullname="Zhou Wang">
<organization></organization>
</author>
<author initials="A.C." surname="Bovik" fullname="Alan Conrad Bovik">
<organization></organization>
</author>
<author initials="H.R." surname="Sheikh" fullname="Hamid Rahim Sheikh">
<organization></organization>
</author>
<author initials="E.P." surname="Simoncelli" fullname="Eero P. Simoncelli">
<organization></organization>
</author>
<date year="2004"/>
</front>
</reference>
<reference anchor="DAALA-GIT" target="http://git.xiph.org/?p=daala.git;a=summary">
<front>
<title>Daala Git Repository</title>
<author >
<organization>Xiph.Org</organization>
</author>
<date year="2015"/>
</front>
</reference>
<reference anchor="AWCY" target="https://arewecompressedyet.com/">
<front>
<title>Are We Compressed Yet?</title>
<author >
<organization>Xiph.Org</organization>
</author>
<date year="2015"/>
</front>
</reference>
<reference anchor="STEAM" target="http://store.steampowered.com/hwsurvey">
<front>
<title>Steam Hardware & Software Survey: June 2015</title>
<author >
<organization>Valve Corporation</organization>
</author>
<date year="2015" month="June"/>
</front>
</reference>
<reference anchor="L1100" target="http://phenix.int-evry.fr/jct/">
<front>
<title>Common test conditions and software reference configurations</title>
<author initials="F." surname="Bossen" fullname="Frank Bossen">
<organization></organization>
</author>
<date year="2013"/>
</front>
<seriesInfo name="JCTVC" value="L1100"/>
</reference>
<reference anchor="COMPARECODECS" target="http://compare-codecs.appspot.com/">
<front>
<title>Compare Codecs</title>
<author initials="H." surname="Alvestrand" fullname="Harald Alvestrand">
<organization></organization>
</author>
<date year="2015"/>
</front>
</reference>
<reference anchor="DERFVIDEO" target="https://media.xiph.org/video/derf/">
<front>
<title>Xiph.org Video Test Media</title>
<author initials="T." surname="Terriberry" fullname="Timothy Terriberry">
<organization></organization>
</author>
<date year="n.d."/>
</front>
</reference>
<reference anchor="TESTSEQUENCES" target="https://people.xiph.org/~tdaede/sets/">
<front>
<title>Test Sets</title>
<author initials="T." surname="Daede" fullname="Thomas Daede">
<organization></organization>
</author>
<date year="n.d."/>
</front>
</reference>
<reference anchor="CIEDE2000" target="http://dx.doi.org/10.1155/2012/273723">
<front>
<title>Color Image Quality Assessment Based on CIEDE2000</title>
<author initials="Y." surname="Yang" fullname="Yang Yang">
<organization></organization>
</author>
<author initials="J." surname="Ming" fullname="Jun Ming">
<organization></organization>
</author>
<author initials="N." surname="Yu" fullname="Nenghai Yu">
<organization></organization>
</author>
<date year="2012"/>
</front>
</reference>
<reference anchor="VMAF" target="http://ieeexplore.ieee.org/xpl/login.jsp?tp=&arnumber=7351097">
<front>
<title>Challenges in cloud based ingest and encoding for high quality streaming media</title>
<author initials="A." surname="Aaron" fullname="Anne Aaron">
<organization></organization>
</author>
<author initials="Z." surname="Li" fullname="Zhi Li">
<organization></organization>
</author>
<author initials="M." surname="Manohara" fullname="Megha Manohara">
<organization></organization>
</author>
<author initials="J.Y." surname="Lin" fullname="Joe Yuchieh Lin">
<organization></organization>
</author>
<author initials="E.C." surname="Wu" fullname="Eddy Chi-Hao Wu">
<organization></organization>
</author>
<author initials="C.J." surname="Kuo" fullname="C.-C Jay Kuo">
<organization></organization>
</author>
<date year="2015"/>
</front>
</reference>
<reference anchor="BT500" target="https://www.itu.int/dms_pubrec/itu-r/rec/bt/R-REC-BT.500-13-201201-I!!PDF-E.pdf">
<front>
<title>Recommendation ITU-R BT.500-13</title>
<author >
<organization>ITU-R</organization>
</author>
<date year="2012"/>
</front>
</reference>
</references>
</back>
</rfc>
| PAFTECH AB 2003-2026 | 2026-04-24 04:20:43 |