One document matched: draft-ietf-netvc-testing-00.xml

<?xml version="1.0" encoding="UTF-8"?>
  <?xml-stylesheet type="text/xsl" href="rfc2629.xslt" ?>
  <!-- generated by https://github.com/cabo/kramdown-rfc2629 version 1.0.20 -->

<!DOCTYPE rfc SYSTEM "rfc2629.dtd" [
]>

<rfc ipr="trust200902" docName="draft-ietf-netvc-testing-00" category="info">

<?rfc toc="yes"?>
<?rfc sortrefs="yes"?>
<?rfc symrefs="yes"?>

  <front>
    <title>Video Codec Testing and Quality Measurement</title>

    <author initials="T." surname="Daede" fullname="Thomas Daede">
      <organization>Mozilla</organization>
      <address>
        <email>tdaede@mozilla.com</email>
      </address>
    </author>

    <date year="2015" month="November" day="30"/>

    <area>RAI</area>
    
    <keyword>Internet-Draft</keyword>

    <abstract>


<t>This document describes guidelines and procedures for evaluating an internet video codec specified at the IETF. This covers subjective and objective tests, test conditions, and materials used for the test.</t>



    </abstract>


  </front>

  <middle>


<section anchor="introduction" title="Introduction">

<t>When developing an internet video codec, changes and additions to the codec need to be decided based on their performance tradeoffs. In addition, measurements are needed to determine when the codec has met its performance goals. This document specifies how the tests are to be carried about to ensure valid comparisons and good decisions.</t>

</section>
<section anchor="subjective-metrics" title="Subjective Metrics">

<t>Subjective testing is the preferable method of testing video codecs.</t>

<t>Because the IETF does not have testing resources of its own, it has to rely on the resources of its participants. For this reason, even if the group agrees that a particular test is important, if no one volunteers to do it, or if volunteers do not complete it in a timely fashion, then that test should be discarded.  This ensures that only important tests be done in particular, the tests that are important to participants.</t>

<section anchor="still-image-pair-comparison" title="Still Image Pair Comparison">

<t>A simple way to determine superiority of one compressed image over another is to visually compare two compressed images, and have the viewer judge which one has a higher quality. This is mainly used for rapid comparisons during development. For this test, the two compressed images should have similar compressed file sizes, with one image being no more than 5% larger than the other. In addition, at least 5 different images should be compared.</t>

</section>
</section>
<section anchor="objective-metrics" title="Objective Metrics">

<t>Objective metrics are used in place of subjective metrics for easy and repeatable experiments. Most objective metrics have been designed to correlate with subjective scores.</t>

<t>The following descriptions give an overview of the operation of each of the metrics. Because implementation details can sometimes vary, the exact implementation is specified in C in the Daala tools repository <xref target="DAALA-GIT"/>.</t>

<t>All of the metrics described in this document are to be applied to the luma plane only. In addition, they are single frame metrics. When applied to the video, the scores of each frame are averaged to create the final score.</t>

<t>Codecs are allowed to internally use downsampling, but must include a normative upsampler, so that the metrics run at the same resolution as the source video. In addition, some metrics, such as PSNR and FASTSSIM, have poor behavior on downsampled images, so it must be noted in test results if downsampling is in effect.</t>

<section anchor="psnr" title="PSNR">

<t>PSNR is a traditional signal quality metric, measured in decibels. It is directly drived from mean square error (MSE), or its square root (RMSE). The formula used is:</t>

<t>20 * log10 ( MAX / RMSE )</t>

<t>or, equivalently:</t>

<t>10 * log10 ( MAX^2 / MSE )</t>

<t>which is the method used in the dump_psnr.c reference implementation.</t>

</section>
<section anchor="psnr-hvs-m" title="PSNR-HVS-M">

<t>The PSNR-HVS metric performs a DCT transform of 8x8 blocks of the image, weights the coefficients, and then calculates the PSNR of those coefficients. Several different sets of weights have been considered. <xref target="PSNRHVS"/> The weights used by the dump_pnsrhvs.c tool in the Daala repository have been found to be the best match to real MOS scores.</t>

</section>
<section anchor="ssim" title="SSIM">

<t>SSIM (Structural Similarity Image Metric) is a still image quality metric introduced in 2004 <xref target="SSIM"/>. It computes a score for each individual pixel, using a window of neighboring pixels. These scores can then be averaged to produce a global score for the entire image. The original paper produces scores ranging between 0 and 1.</t>

<t>For the metric to appear more linear on BD-rate curves, the score is converted into a nonlinear decibel scale:</t>

<t>-10 * log10 (1 - SSIM)</t>

</section>
<section anchor="fast-multi-scale-ssim" title="Fast Multi-Scale SSIM">

<t>Multi-Scale SSIM is SSIM extended to multiple window sizes <xref target="MSSSIM"/>. This is implemented in the Fast implementation by downscaling the image a number of times, and computing SSIM over the same number of pixels, then averaging the SSIM scores together <xref target="FASTSSIM"/>. The final score is converted to decibels in the same manner as SSIM.</t>

</section>
</section>
<section anchor="comparing-and-interpreting-results" title="Comparing and Interpreting Results">

<section anchor="graphing" title="Graphing">

<t>When displayed on a graph, bitrate is shown on the X axis, and the quality metric is on the Y axis. For clarity, the X axis bitrate is always graphed in the log domain. The Y axis metric should also be chosen so that the graph is approximately linear. For metrics such as PSNR and PSNR-HVS, the metric result is already in the log domain and is left as-is. SSIM and FASTSSIM, on the other hand, return a result between 0 and 1. To create more linear graphs, this result is converted to a value in decibels:</t>

<t>-1 * log10 ( 1 - SSIM )</t>

</section>
<section anchor="bjontegaard" title="Bjontegaard">

<t>The Bjontegaard rate difference, also known as BD-rate, allows the comparison of two different codecs based on a metric. This is commonly done by fitting a curve to each set of data points on the plot of bitrate versus metric score, and then computing the difference in area between each of the curves. A cubic polynomial fit is common, but will be overconstrained with more than four samples. For higher accuracy, at least 10 samples and a linear piecewise fit should be used. In addition, if using a truncated BD-rate curve, there should be at least 4 samples within the point of interest.</t>

</section>
<section anchor="ranges" title="Ranges">

<t>The curve is split into three regions, for low, medium, and high bitrate. The ranges are defined as follows:</t>

<t><list style="symbols">
  <t>Low bitrate: 0.005 - 0.02 bpp</t>
  <t>Medium bitrate: 0.02 - 0.06 bpp</t>
  <t>High bitrate: 0.06 - 0.2 bpp</t>
</list></t>

<t>Bitrate can be calculated from bits per pixel (bpp) as follows:</t>

<t>bitrate = bpp * width * height * framerate</t>

</section>
</section>
<section anchor="test-sequences" title="Test Sequences">

<section anchor="sources" title="Sources">

<t>Lossless test clips are preferred for most tests, because the structure of compression artifacts in already-compressed clips may introduce extra noise in the test results. However, a large amount of content on the internet needs to be recompressed at least once, so some sources of this nature are useful. The encoder should run at the same bit depth as the original source. In addition, metrics need to support operation at high bit depth. If one or more codecs in a comparison do not support high bit depth, sources need to be converted once before entering the encoder.</t>

<t>The JCT-VC standards organization includes a set of standard test clips for video codec testing, and parameters to run the clips with <xref target="L1100"/>. These clips are not publicly available, but are very useful for comparing to published results.</t>

<t>Xiph publishes a variety of test clips collected from various sources.</t>

<t>The Blender Open Movie projects provide a large test base of lossless cinematic test material. The lossless sources are available, hosted on Xiph.</t>

</section>
<section anchor="test-sets" title="Test Sets">

<t>Sources are divided into several categories to test different scenarios the codec will be required to operate in. For easier comaprison, all videos in each set should have the same color subsampling, same resolution, and same number of frames. In addition, all test videos must be publicly available for testing use, to allow for reproducibility of results.</t>

<t><list style="symbols">
  <t>Still images are useful when comparing intra coding performance. Xiph.org has four sets of lossless, one megapixel images that have been converted into YUV 4:2:0 format. There are four sets that can be used:
  <list style="symbols">
      <t>subset1 (50 images)</t>
      <t>subset2 (50 images)</t>
      <t>subset3 (1000 images)</t>
      <t>subset4 (1000 images)</t>
    </list></t>
  <t>video-hd-2, a set that consists of the following 1920x1080 clips from <xref target="DERFVIDEO"/>, cropped to 50 frames (and converted to 4:2:0 if necessary)
  <list style="symbols">
      <t>aspen</t>
      <t>blue_sky</t>
      <t>crowd_run</t>
      <t>ducks_take_off</t>
      <t>factory</t>
      <t>life</t>
      <t>old_town_cross</t>
      <t>park_joy</t>
      <t>pedestrian_area</t>
      <t>red_kayak</t>
      <t>riverbed</t>
      <t>rush_hour</t>
      <t>station2</t>
    </list></t>
  <t>A video conferencing test set, with 1280x720 content at 60 frames per second. Unlike other sets, the videos in this set are 10 seconds long.
  <list style="symbols">
      <t>TBD</t>
    </list></t>
  <t>Game streaming content: 1920x1080, 60 frames per second, 4:2:0 chroma subsampling. 1080p is chosen as it is currently the most common gaming monitor resolution <xref target="STEAM"/>. All clips should be two seconds long.
  <list style="symbols">
      <t>TBD</t>
    </list></t>
  <t>Screensharing content is low framerate, high resolution content typical of a computer desktop.
  <list style="symbols">
      <t>screenshots - desktop screenshots of various resolutions, with 4:2:0 subsampling</t>
      <t>Video sets TBD</t>
    </list></t>
</list></t>

</section>
<section anchor="operating-points" title="Operating Points">

<t>Two operating modes are defined. High latency is intended for on demand streaming, one-to-many live streaming, and stored video. Low latency is intended for videoconferencing and remote access.</t>

<section anchor="high-latency" title="High Latency">

<t>The encoder should be run at the best quality mode available, using the mode that will provide the best quality per bitrate (VBR or constant quality mode). Lookahead and/or two-pass are allowed, if supported. One parameter is provided to adjust bitrate, but the units are arbitrary.  Example configurations follow:</t>

<t><list style="symbols">
  <t>x264: –crf=x</t>
  <t>x265: –crf=x</t>
  <t>daala: -v=x</t>
  <t>libvpx: –codec=vp9 –end-usage=q –cq-level=x -lag-in-frames=25 -auto-alt-ref=1</t>
</list></t>

</section>
<section anchor="unconstrained-low-latency" title="Unconstrained Low Latency">

<t>The encoder should be run at the best quality mode available, using the mode that will provide the best quality per bitrate (VBR or constant quality mode), but no frame delay, buffering, or lookahead is allowed. One parameter is provided to adjust bitrate, but the units are arbitrary. Example configurations follow:</t>

<t><list style="symbols">
  <t>x264: –crf-x –tune zerolatency</t>
  <t>x265: –crf=x –tune zerolatency</t>
  <t>daala: -v=x</t>
  <t>libvpx: –codec=vp9 –end-usage=q –cq-level=x -lag-in-frames=0 -auto-alt-ref=0</t>
</list></t>

</section>
<section anchor="constrained-low-latency" title="Constrained Low Latency">

<t>The encoder is given one parameter, which is absolute bitrate. No frame delay, buffering, or lookahead is allowed. The maximum achieved bitrate deviation from the supplied parameter is determined by a buffer model:</t>

<t><list style="symbols">
  <t>The buffer starts out empty.</t>
  <t>After each frame is encoded, the buffer is filled by the number of bits spent for the frame.</t>
  <t>The buffer is then emptied by (bitrate * frame duration) bits.</t>
  <t>The buffer fill level is checked. If it is over the limit, the test is considered a failure.</t>
</list></t>

<t>The buffer size limit is defined by the bitrate target * 0.3 seconds.</t>

</section>
</section>
</section>
<section anchor="automation" title="Automation">

<t>Frequent objective comparisons are extremely beneficial while developing a new codec. Several tools exist in order to automate the process of objective comparisons. The Compare-Codecs tool allows BD-rate curves to be generated for a wide variety of codecs <xref target="COMPARECODECS"/>. The Daala source repository contains a set of scripts that can be used to automate the various metrics used. In addition, these scripts can be run automatically utilizing distributed computer for fast results <xref target="AWCY"/>.</t>

</section>


  </middle>

  <back>


    <references title='Informative References'>

<reference anchor="PSNRHVS" >
  <front>
    <title>A New Full-Reference Quality Metrics Based on HVS</title>
    <author initials="K." surname="Egiazarian" fullname="Karen Egiazarian">
      <organization></organization>
    </author>
    <author initials="J." surname="Astola" fullname="Jaakko Astola">
      <organization></organization>
    </author>
    <author initials="N." surname="Ponomarenko" fullname="Nikolay Ponomarenko">
      <organization></organization>
    </author>
    <author initials="V." surname="Lukin" fullname="Vladimir Lukin">
      <organization></organization>
    </author>
    <author initials="F." surname="Battisti" fullname="Federica Battisti">
      <organization></organization>
    </author>
    <author initials="M." surname="Carli" fullname="Marco Carli">
      <organization></organization>
    </author>
    <date year="2002"/>
  </front>
</reference>
<reference anchor="FASTSSIM" target="http://live.ece.utexas.edu/publications/2011/chen_rtip_2011.pdf">
  <front>
    <title>Fast structural similarity index algorithm</title>
    <author initials="M." surname="Chen" fullname="Ming-Jun Chen">
      <organization></organization>
    </author>
    <author initials="A.C." surname="Bovik" fullname="Alan Conrad Bovik">
      <organization></organization>
    </author>
    <date year="2010"/>
  </front>
</reference>
<reference anchor="MSSSIM" target="http://www.cns.nyu.edu/~zwang/files/papers/msssim.pdf">
  <front>
    <title>Multi-Scale Structural Similarity for Image Quality Assessment</title>
    <author initials="Z." surname="Wang" fullname="Zhou Wang">
      <organization></organization>
    </author>
    <author initials="E.P." surname="Simoncelli" fullname="Eero P. Simoncelli">
      <organization></organization>
    </author>
    <author initials="A.C." surname="Bovik" fullname="Alan Conrad Bovik">
      <organization></organization>
    </author>
    <date year="n.d."/>
  </front>
</reference>
<reference anchor="SSIM" target="http://www.cns.nyu.edu/pub/eero/wang03-reprint.pdf">
  <front>
    <title>Image Quality Assessment: From Error Visibility to Structural Similarity</title>
    <author initials="Z." surname="Wang" fullname="Zhou Wang">
      <organization></organization>
    </author>
    <author initials="A.C." surname="Bovik" fullname="Alan Conrad Bovik">
      <organization></organization>
    </author>
    <author initials="H.R." surname="Sheikh" fullname="Hamid Rahim Sheikh">
      <organization></organization>
    </author>
    <author initials="E.P." surname="Simoncelli" fullname="Eero P. Simoncelli">
      <organization></organization>
    </author>
    <date year="2004"/>
  </front>
</reference>
<reference anchor="DAALA-GIT" target="http://git.xiph.org/?p=daala.git;a=summary">
  <front>
    <title>Daala Git Repository</title>
    <author >
      <organization>Xiph.Org</organization>
    </author>
    <date year="2015"/>
  </front>
</reference>
<reference anchor="AWCY" target="https://arewecompressedyet.com/">
  <front>
    <title>Are We Compressed Yet?</title>
    <author >
      <organization>Xiph.Org</organization>
    </author>
    <date year="2015"/>
  </front>
</reference>
<reference anchor="STEAM" target="http://store.steampowered.com/hwsurvey">
  <front>
    <title>Steam Hardware & Software Survey: June 2015</title>
    <author >
      <organization>Valve Corporation</organization>
    </author>
    <date year="2015" month="June"/>
  </front>
</reference>
<reference anchor="L1100" target="http://phenix.int-evry.fr/jct/">
  <front>
    <title>Common test conditions and software reference configurations</title>
    <author initials="F." surname="Bossen" fullname="Frank Bossen">
      <organization></organization>
    </author>
    <date year="2013"/>
  </front>
  <seriesInfo name="JCTVC" value="L1100"/>
</reference>
<reference anchor="COMPARECODECS" target="http://compare-codecs.appspot.com/">
  <front>
    <title>Compare Codecs</title>
    <author initials="H." surname="Alvestrand" fullname="Harald Alvestrand">
      <organization></organization>
    </author>
    <date year="2015"/>
  </front>
</reference>
<reference anchor="DERFVIDEO" target="https://media.xiph.org/video/derf/">
  <front>
    <title>Xiph.org Video Test Media</title>
    <author initials="T." surname="Terriberry" fullname="Timothy Terriberry">
      <organization></organization>
    </author>
    <date year="n.d."/>
  </front>
</reference>


    </references>



  </back>
</rfc>