One document matched: draft-ietf-avtext-avpf-ccm-layered-00.xml
<?xml version="1.0" encoding="US-ASCII"?>
<!DOCTYPE rfc SYSTEM "rfc2629.dtd" [
<!ENTITY vp9 SYSTEM "http://xml.resource.org/public/rfc/bibxml3/reference.I-D.grange-vp9-bitstream.xml">
]>
<?rfc toc="yes"?>
<?rfc tocompact="yes"?>
<?rfc tocdepth="3"?>
<?rfc tocindent="yes"?>
<?rfc symrefs="yes"?>
<?rfc sortrefs="yes"?>
<?rfc comments="yes"?>
<?rfc inline="yes"?>
<?rfc compact="yes"?>
<?rfc subcompact="no"?>
<rfc category="std" docName="draft-ietf-avtext-avpf-ccm-layered-00" ipr="trust200902" updates="5104">
<front>
<title abbrev="CCM-Layered">Using Codec Control Messages in the RTP Audio-Visual Profile with Feedback with Layered Codecs</title>
<author fullname="Stephan Wenger" initials="S." surname="Wenger">
<organization>Vidyo, Inc.</organization>
<address>
<postal>
<street></street>
<!-- Reorder these if your country does things differently -->
<city></city>
<region></region>
<code></code>
<country></country>
</postal>
<phone></phone>
<email>stewe@stewe.org</email>
<!-- uri and facsimile elements may also be added -->
</address>
</author>
<author fullname="Jonathan Lennox" initials="J." surname="Lennox">
<organization>Vidyo, Inc.</organization>
<address>
<postal>
<street></street>
<!-- Reorder these if your country does things differently -->
<city></city>
<region></region>
<code></code>
<country></country>
</postal>
<phone></phone>
<email>jonathan@vidyo.com</email>
<!-- uri and facsimile elements may also be added -->
</address>
</author>
<author fullname="Bo Burman" initials="B." surname="Burman">
<organization>Ericsson</organization>
<address>
<postal>
<street>Kistavagen 25</street>
<city>SE - 164 80 Kista</city>
<region></region>
<code></code>
<country>Sweden</country>
</postal>
<phone></phone>
<facsimile></facsimile>
<email>bo.burman@ericsson.com</email>
<uri></uri>
</address>
</author>
<author fullname="Magnus Westerlund" initials="M." surname="Westerlund">
<organization>Ericsson</organization>
<address>
<postal>
<street>Farogatan 6</street>
<city>SE- 164 80 Kista</city>
<region></region>
<code></code>
<country>Sweden</country>
</postal>
<phone>+46107148287</phone>
<facsimile></facsimile>
<email>magnus.westerlund@ericsson.com</email>
</address>
</author>
<date day="26" month="April" year="2016" />
<abstract>
<t>This document fixes a shortcoming in the specification language of the Codec Control Message Full Intra Request (FIR) as defined in RFC5104 when using with layered codecs. In particular, a Decoder Refresh Point needs to be sent by a media sender when a FIR is received on any layer of the layered bitstream, regardless on whether those layers are being sent in a single or in multiple RTP flows. The other payload-specific feedback messages defined in RFC 5104 and RFC 4585 as updated by RFC 5506 have also been analyzed, and no corresponding shortcomings have been found.
</t>
</abstract>
</front>
<middle>
<section title="Introduction and Problem Statement">
<t> <xref target="RFC4585">RFC 4585</xref> and <xref target="RFC5104">RFC 5104</xref> specify a number of payload-specific feedback messages which a media receiver can use to inform a media sender of certain conditions, or make certain requests. The feedback messages are being sent as RTCP receiver reports, and RFC 4585 specifies timing rules that make the use of those messages practical for time-sensitive codec control. </t>
<t> Since the time those RFCs were developed, layered codecs have gained in popularity and deployment. Layered codecs use multiple sub-bitstreams called layers to represent the content in different fidelities. Depending on the media codec and its RTP payload format in use, single layers or groups of layers may be sent in their own RTP streams (in MRST or MRMT mode as defined in <xref target="RFC7656">RFC 7656</xref>), or multiplexed (using media-codec specific multiplexing mechanisms) in a single RTP stream (SRST mode as defined in <xref target="RFC7656">RFC 7656</xref>). The dependency relationship between layers forms a directed graph, with the base layer at the root. Enhancement layers depend on the base layer and potentially on other enhancement layers, and the target layer and all layers it depends on have to be decoded jointly in order to re-create the uncompressed media signal at the fidelity of the target layer. </t>
<t> Implementation experience has shown that the Full Intra Request command as defined in <xref target="RFC5104">RFC 5104</xref> is underspecified when used with layered codecs and when more than one RTP stream is used to transport the layers of a layered bitstream at a given fidelity. In particular, from the <xref target="RFC5104">RFC 5104</xref> specification language it is not clear whether an FIR received for only a single RTP stream of multiple RTP streams covering the same layered bitstream necessarily triggers the sending of a Decoder Refresh Point (as defined in <xref target="RFC5104">RFC 5104</xref> section 2.2) for all layers, or only for the layer which is transported in the RTP stream which the FIR request is associated with. </t>
<t> This document fixes this shortcoming by:
<list style="letters">
<t>Updating the definition of the Decoder Refresh Point (as defined in <xref target="RFC5104">RFC 5104</xref> section 2.2) to cover layered codecs, in line with the corresponding definitions used in a popular layered codec format, namely <xref target="H.264">H.264/SVC</xref>. Specifically, a decoder refresh point, in conjunction with layered codecs, resets the state of the whole decoder, which implies that it includes hard or gradual single-layer decoder refresh for all layers; </t>
<t>Requiring that, when a media sender receives a Full Intra Request over the RTCP stream associated with any of the RTP streams over which a part of the layered bitstream is transported, to send a Decoder Refresh Point; </t>
<t>Require that a media receiver sends the FIR on the RTCP stream associated with the base layer (the option of receiving FIR on enhancement layer-associated RTCP stream as specified in point b) above is kept for backward compatibility); and </t>
<t>Providing guidance on how to detect that a layered codec is in use for which the above rules apply. </t>
</list></t>
<t> While, clearly, the reaction to FIR for layered codecs in <xref target="RFC5104">RFC 5104</xref> and companion documents is underspecified, it appears that this is not the case for any of the other payload-specific codec control messages defined in any of <xref target="RFC4585">RFC 4585</xref>, <xref target="RFC5104">RFC 5104</xref>, or <xref target="RFC5506">RFC 5506</xref>. A brief summary of the analysis that led to this conclusion is also included in this document. </t>
</section>
<section title="Requirements Language">
<t>The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
document are to be interpreted as described in <xref target="RFC2119">RFC 2119</xref>.</t>
</section>
<section anchor="DRP" title="Updated definition of Decoder Refresh Point">
<t>The text below updates the definition of Decoder Refresh Point in section 2.2 of <xref target="RFC5104">RFC 5104</xref>.</t>
<t>Decoder Refresh Point: A bit string, packetized in one or more RTP packets, that
completely resets the decoder to a known state.</t>
<t> Examples for "hard" single layer decoder refresh points are Intra pictures
in <xref target="H.261">H.261</xref>, <xref target="H.263">H.263</xref>, <xref target="MPEG-1">MPEG-1</xref>, <xref target="MPEG-2">MPEG-2</xref>, and <xref target="MPEG-4">MPEG-4</xref>;
Instantaneous Decoder Refresh (IDR) pictures in <xref target="H.264">H.264</xref>, and <xref target="H.265">H.265</xref>;
and Keyframes in <xref target="RFC6386">VP8</xref> and <xref target="I-D.grange-vp9-bitstream">VP9</xref>.
"Gradual" decoder refresh points may also be used; see for
example <xref target="H.264">H.264</xref>. While both "hard" and "gradual" decoder
refresh points are acceptable in the scope of this
specification, in most cases the user experience will benefit
from using a "hard" decoder refresh point.</t>
<t> A decoder refresh point also contains all header information
above the syntactical level of the picture layer (or equivalent, depending on the video
compression standard) that is conveyed in-band. In <xref target="H.264">H.264</xref>, for
example, a decoder refresh point contains parameter set
Network Adaptation Layer (NAL) units that generate parameter
sets necessary for the decoding of the following slice/data
partition NAL units (and that are not conveyed out of band).</t>
<t> When a layered codec is in use, the above definition (and,
in particular, the requirement to COMPLETELY reset the
decoder to a known state) implies that the decoder refresh
point includes hard or gradual single layer decoder refresh
points for all layers. </t>
</section>
<section title="Full Intra Request for Layered Codecs">
<t> When a media receiver or middlebox has decided to send a FIR command (based on the guidance provided in Section 4.3.1 of <xref target="RFC5104">RFC 5104</xref>, it MUST do so in the RTCP stream related to the forward RTP stream that carries the base layer of the layered bitstream, and the Feedback Control Information (FCI, and in particular the SSRC field therein) MUST also refer to the forward RTP stream that carries the base layer.</t>
<t> When a Full Intra Request Command is received by the designated media sender in the RTCP stream associated with any of the RTP streams in which any layer of a layered bitstream are sent, the designated media sender MUST send a <xref target="DRP">Decoder Refresh Point</xref> as defined above at its earliest opportunity. The requirements related to congestion control on the forward RTP streams as specified in sections 3.5.1.5 of <xref target="RFC5104">RFC 5104</xref> apply for the RTP streams both in isolation and combined.</t>
<t> Note: the requirement to react to FIR commands associated with enhancement layers is included for robustness and backward compatibility reasons.</t>
</section>
<section title="Identifying the use of Layered Codecs (Informative)">
<t> The above modifications to RFC 5104 unambiguously define how to deal with FIR when layered bitstreams are in use. However, it is surprisingly difficult to identify this situation. In general, it is expected that implementers know when layered coding (in its commonly understood sense: with inter-layer prediction between pyramided-arranged layers) is in use and when not, and can therefore implement the above updates to RFC 5104 correctly. However, there are use cases of the use of layered codecs that may be viewed as somewhat exotic today but clearly are supported by the video coding syntax, in which the above rules would lead to suboptimal system behavior. Nothing would break, and there would not be an interop failure, but the user experience may suffer through the sending or receiving of Decoder Refresh Points at times or on parts of the bitstream that are unnecessary from a user experience viewpoint. Therefore, this informative section is included that provides the current understanding of when a layered codec is in use and when not.</t>
<t> The key observation made here is that the RTP payload format negotiated for the RTP streams, in isolation, is not necessarily an indicator for the use of layering. Some layered codecs (including H.264/SVC) can form decodable bitstreams including only (one or more) enhancement layers, without the base layer, effectively creating simulcastable sub-bitstreams in a scalable bitstream that does not take advantage of inter-layer prediction. In such a scenario, it is potentially (though not necessarily) unnecessary--or even counter-productive--to send a decoder refresh point on all RTP streams using that payload format and SSRC. </t>
<t> One good indication of the likely use of layering with interlayer prediction is when the various RTP streams are "bound" together on the signaling level. In an SDP environment, this would be the case if they are marked as being dependent from each other using the grouping framework <xref target="RFC4588">RFC 4588</xref> and the layer dependency <xref target="RFC5583">RFC 5583</xref>. Conversely, one good indication of the use of simulcast is when simulcasting is explicitly being signaled, for example through the use of
<xref target="I-D.ietf-mmusic-sdp-simulcast" />,
except when simulcast stream identifiers are explicitly defined as dependent according to
<xref target="I-D.ietf-mmusic-rid" />. </t>
</section>
<section title="Layered Codecs and non-FIR codec control messages (Informative)">
<t>Between them, <xref target="RFC4585">RFC 4585</xref> (as updated by <xref target="RFC5506">RFC 5506</xref>) and <xref target="RFC5104">RFC 5104</xref> define a total of seven Payload-specific Feedback messages. For the FIR command message, guidance has been provided above. In this section, some information is provided with respect to the remaining six codec control messages.</t>
<section title="Picture Loss Indication (PLI)">
<t>PLI is defined in <xref target="RFC4585">RFC 4585</xref> section 6.3.1. The prudent response to a PLI message received for an enhancement layer is to "repair" (through whatever source-coding specific means) that enhancement layer and all dependent enhancement layers, but not the reference layer(s) used by the enhancement layer for which the PLI was received. The encoder can figure out by itself what constitutes a dependent enhancement layer and does not need help from the system stack in doing so. Insofar, there is nothing that needs to be specified herein. </t>
</section>
<section title="Slice Loss Indication (SLI)">
<t>SLI is defined in <xref target="RFC4585">RFC 4585</xref> section 6.3.2. The authors' current understanding is that the prudent response to a SLI message received for an enhancement layer is to "repair" (through whatever source-coding specific means) the affected spatial area of that enhancement layer and all dependent enhancement layers, but not the reference layers used by the enhancement layer for which the SLI was received. The encoder can figure out by itself what constitutes a dependent enhancement layer and does not need help from the system stack in doing so. Insofar, there is nothing that needs to be specified herein. SLI has seen very little implementation and, as far as it is known, none in conjunction with layered systems. </t>
</section>
<section title="Reference Picture Selection Indication (RPSI)" anchor="RPSI">
<t>RPSI is defined in <xref target="RFC4585">RFC 4585</xref> section 6.3.3. While a technical equivalent of RPSI has been in use with non-layered systems for many years, no implementations are known in conjunction of layered codecs. The authors' current understanding is that the reception of an RPSI message on any layer forces the encoder to "repair" the bitstream on that layer and all dependent layers without the need of any system-provided guidance. Insofar, RPSI should work without further need for specification language. </t>
</section>
<section title="Temporal-Spatial Trade-off Request and Notification (TSTR/TSTN)">
<t>TSTN/TSTR are defined in <xref target="RFC5104">RFC 5104</xref> section 4.3.2 and 4.3.3, respectively. The TSTR request allows to communicate (typically user-interface-obtained) guidance of the preferred trade-off between spatial quality and frame rate. A technical equivalent of TSTN/TSTR has seen deployment for many years in non-scalable systems. </t>
<t> The Temporal-Spatial Trade-off request and notification messages include an SSRC target, which (similarly to FIR) may refer to an RTP stream carrying a base layer, an enhancement layer, or multiple layers. Therefore, the authors' current understanding is that the semantics of the message applies to the layers present in the targeted RTP stream. </t>
<t> It is noted that per-layer TSTR/TSTN is a mechanism that is, in some ways, counterproductive in a system using layered codecs. Given a sufficiently complex layered bitstream layout, a sending system has flexibility in adjusting the spatio/temporal quality balance by adding and removing temporal, spatial, or quality enhancement layers. At present it is unclear whether an allowed (or even recommended) option to the reception of a TSTR is to adjust the bit allocation within the layer(s) present in the addressed RTP stream, or to adjust the layering structure accordingly--which can involve more than just the addressed RTP stream. </t>
<t> Until there is a sufficient critical mass of implementation practice, it is probably prudent for an implementer not to assume either of the two options (or any middleground that may exist between the two), be liberal in accepting TSTR messages, perhaps responding in TSTN indicating "no change," not sending TSTR messages except when operating in SRST mode as defined in <xref target="RFC7656">RFC 7656</xref>, and contribute to the IETF documentation of any implementation requirements that make per-layer TSTR/TSTN useful. </t>
</section>
<section title="H.271 Video Back Channel Message (VBCM)">
<t>VBCM is defined in <xref target="RFC5104">RFC 5104</xref> section 4.3.4. What was said above for <xref target="RPSI">RPSI</xref> applies here as well. </t>
</section>
</section>
<section title="Acknowledgements">
<t>The authors want to thank Mo Zanaty for useful discussions.</t>
</section>
<section title="IANA Considerations">
<t>This memo includes no request to IANA.</t>
</section>
<section title="Security Considerations">
<t>The security considerations of <xref target="RFC4585">RFC 4585</xref> (as updated by <xref target="RFC5506">RFC 5506</xref>) and <xref target="RFC5104">RFC 5104</xref> apply. The clarified response to FIR does not require any updates.</t>
</section>
</middle>
<!-- *****BACK MATTER ***** -->
<back>
<references title="Normative References">
<?rfc include='reference.RFC.2119'?>
<?rfc include='reference.RFC.4585'?>
<?rfc include='reference.RFC.5104'?>
<?rfc include="reference.RFC.5506"?>
</references>
<references title="Informative References">
<?rfc include='reference.RFC.4588'?>
<?rfc include='reference.RFC.5583'?>
<?rfc include='reference.RFC.7656'?>
<?rfc include='reference.RFC.6386'?>
<?rfc include='reference.I-D.ietf-mmusic-sdp-simulcast'?>
<?rfc include='reference.I-D.ietf-mmusic-rid'?>
<!--
<?rfc include 'reference.I-D.grange-vp9-bitstream'?>
-->
&vp9;
<reference anchor="H.261" target="http://handle.itu.int/11.1002/1000/1088">
<front>
<title>ITU-T Rec. H.261: Video codec for audiovisual services at p x 64 kbit/s </title>
<author> <organization>ITU-T</organization> </author>
<date year="1993" />
</front>
</reference>
<reference anchor="H.263" target="http://handle.itu.int/11.1002/1000/7497">
<front>
<title>ITU-T Rec. H.263: Video coding for low bit rate communication</title>
<author> <organization>ITU-T</organization> </author>
<date year="2005" />
</front>
</reference>
<reference anchor="H.264" target="http://handle.itu.int/11.1002/1000/12063">
<front>
<title>ITU-T Rec. H.264: Advanced video coding for generic audiovisual services</title>
<author> <organization>ITU-T</organization> </author>
<date year="2014" />
</front>
</reference>
<reference anchor="H.265" target="http://handle.itu.int/11.1002/1000/12455">
<front>
<title>ITU-T Rec. H.265: High efficiency video coding</title>
<author> <organization>ITU-T</organization> </author>
<date year="2015" />
</front>
</reference>
<reference anchor="MPEG-1">
<front>
<title>ISO/IEC 11172-2:1993 Information technology -- Coding of moving pictures and associated audio for digital storage media at up to about 1,5 Mbit/s -- Part 2: Video</title>
<author> <organization>ISO/IEC</organization> </author>
<date year="1993" />
</front>
</reference>
<reference anchor="MPEG-2">
<front>
<title>ISO/IEC 13818-2:2013 Information technology -- Generic coding of moving pictures and associated audio information -- Part 2: Video</title>
<author> <organization>ISO/IEC</organization> </author>
<date year="2013" />
</front>
</reference>
<reference anchor="MPEG-4">
<front>
<title>ISO/IEC 14496-2:2004 Information technology -- Coding of audio-visual objects -- Part 2: Visual</title>
<author> <organization>ISO/IEC</organization> </author>
<date year="2004" />
</front>
</reference>
</references>
<section title="Change Log">
<t>NOTE TO RFC EDITOR: Please remove this section prior to
publication.</t>
<t> draft-wenger-avtext-avpf-ccm-layered-00-00: initial version </t>
<t> draft-wenger-avtext-avpf-ccm-layered-00-00: resubmit as avtext WG draft per IETF95 and list confirmation by Rachel 4/25/2016 </t>
</section>
</back>
</rfc>
| PAFTECH AB 2003-2026 | 2026-04-24 04:26:21 |