One document matched: draft-ietf-avtext-avpf-ccm-layered-02.xml
<?xml version="1.0" encoding="US-ASCII"?>
<!DOCTYPE rfc SYSTEM "rfc2629.dtd">
<?rfc toc="yes"?>
<?rfc tocompact="yes"?>
<?rfc tocdepth="3"?>
<?rfc tocindent="yes"?>
<?rfc symrefs="yes"?>
<?rfc sortrefs="yes"?>
<?rfc comments="yes"?>
<?rfc inline="yes"?>
<?rfc compact="yes"?>
<?rfc subcompact="no"?>
<rfc category="std" docName="draft-ietf-avtext-avpf-ccm-layered-02"
ipr="trust200902" updates="5104">
<front>
<title abbrev="CCM-Layered">Using Codec Control Messages in the RTP
Audio-Visual Profile with Feedback with Layered Codecs</title>
<author fullname="Stephan Wenger" initials="S." surname="Wenger">
<organization>Vidyo, Inc.</organization>
<address>
<postal>
<street/>
<!-- Reorder these if your country does things differently -->
<city/>
<region/>
<code/>
<country/>
</postal>
<phone/>
<email>stewe@stewe.org</email>
<!-- uri and facsimile elements may also be added -->
</address>
</author>
<author fullname="Jonathan Lennox" initials="J." surname="Lennox">
<organization>Vidyo, Inc.</organization>
<address>
<postal>
<street/>
<!-- Reorder these if your country does things differently -->
<city/>
<region/>
<code/>
<country/>
</postal>
<phone/>
<email>jonathan@vidyo.com</email>
<!-- uri and facsimile elements may also be added -->
</address>
</author>
<author fullname="Bo Burman" initials="B." surname="Burman">
<organization>Ericsson</organization>
<address>
<postal>
<street>Kistavagen 25</street>
<city>SE - 164 80 Kista</city>
<region/>
<code/>
<country>Sweden</country>
</postal>
<phone/>
<facsimile/>
<email>bo.burman@ericsson.com</email>
<uri/>
</address>
</author>
<author fullname="Magnus Westerlund" initials="M." surname="Westerlund">
<organization>Ericsson</organization>
<address>
<postal>
<street>Farogatan 2</street>
<city>SE- 164 80 Kista</city>
<region/>
<code/>
<country>Sweden</country>
</postal>
<phone>+46107148287</phone>
<facsimile/>
<email>magnus.westerlund@ericsson.com</email>
</address>
</author>
<date day="22" month="September" year="2016"/>
<abstract>
<t>This document updates RFC5104 by fixing a shortcoming in the
specification language of the Codec Control Message Full Intra Request
(FIR) as defined in RFC5104 when using it with layered codecs. In
particular, a Decoder Refresh Point needs to be sent by a media sender
when a FIR is received on any layer of the layered bitstream, regardless
on whether those layers are being sent in a single or in multiple RTP
flows. The other payload-specific feedback messages defined in RFC 5104
and RFC 4585 as updated by RFC 5506 have also been analyzed, and no
corresponding shortcomings have been found.</t>
</abstract>
</front>
<middle>
<section title="Introduction and Problem Statement">
<t><xref target="RFC4585">Extended RTP Profile for Real-time Transport
Control Protocol (RTCP)-Based Feedback (RTP/AVPF)</xref> and <xref
target="RFC5104">Codec Control Messages in the RTP Audio-Visual Profile
with Feedback (AVPF)</xref> specify a number of payload-specific
feedback messages which a media receiver can use to inform a media
sender of certain conditions, or make certain requests. The feedback
messages are being sent as RTCP receiver reports, and RFC 4585 specifies
timing rules that make the use of those messages practical for
time-sensitive codec control.</t>
<t>Since the time those RFCs were developed, layered codecs have gained
in popularity and deployment. Layered codecs use multiple sub-bitstreams
called layers to represent the content in different fidelities.
Depending on the media codec and its RTP payload format in use, single
layers or groups of layers may be sent in their own RTP streams (in MRST
or MRMT mode as defined in <xref target="RFC7656">A Taxonomy of
Semantics and Mechanisms for Real-Time Transport Protocol (RTP)
Sources</xref>), or multiplexed (using media-codec specific multiplexing
mechanisms) in a single RTP stream (SRST mode as defined in <xref
target="RFC7656"/>). The dependency relationship between layers forms a
directed graph, with the base layer at the root. Enhancement layers
depend on the base layer and potentially on other enhancement layers,
and the target layer and all layers it depends on have to be decoded
jointly in order to re-create the uncompressed media signal at the
fidelity of the target layer.</t>
<t>Implementation experience has shown that the Full Intra Request
command as defined in <xref target="RFC5104"/> is underspecified when
used with layered codecs and when more than one RTP stream is used to
transport the layers of a layered bitstream at a given fidelity. In
particular, from the <xref target="RFC5104"/> specification language it
is not clear whether an FIR received for only a single RTP stream of
multiple RTP streams covering the same layered bitstream necessarily
triggers the sending of a Decoder Refresh Point (as defined in <xref
target="RFC5104"/> section 2.2) for all layers, or only for the layer
which is transported in the RTP stream which the FIR request is
associated with.</t>
<t>This document fixes this shortcoming by: <list style="letters">
<t>Updating the definition of the Decoder Refresh Point (as defined
in <xref target="RFC5104"/> section 2.2) to cover layered codecs, in
line with the corresponding definitions used in a popular layered
codec format, namely <xref target="H.264">H.264/SVC</xref>.
Specifically, a decoder refresh point, in conjunction with layered
codecs, resets the state of the whole decoder, which implies that it
includes hard or gradual single-layer decoder refresh for all
layers;</t>
<t>Requiring that, when a media sender receives a Full Intra Request
over the RTCP stream associated with any of the RTP streams over
which a part of the layered bitstream is transported, to send a
Decoder Refresh Point;</t>
<t>Require that a media receiver sends the FIR on the RTCP stream
associated with the base layer (the option of receiving FIR on
enhancement layer-associated RTCP stream as specified in point b)
above is kept for backward compatibility); and</t>
<t>Providing guidance on how to detect that a layered codec is in
use for which the above rules apply.</t>
</list></t>
<t>While, clearly, the reaction to FIR for layered codecs in <xref
target="RFC5104"/> and companion documents is underspecified, it appears
that this is not the case for any of the other payload-specific codec
control messages defined in any of <xref target="RFC4585"/>, <xref
target="RFC5104"/>. A brief summary of the analysis that led to this
conclusion is also included in this document.</t>
</section>
<section title="Requirements Language">
<t>The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
document are to be interpreted as described in <xref
target="RFC2119">RFC 2119</xref>.</t>
</section>
<section anchor="DRP" title="Updated definition of Decoder Refresh Point">
<t>The text below updates the definition of Decoder Refresh Point in
section 2.2 of <xref target="RFC5104"/>.</t>
<t>Decoder Refresh Point: A bit string, packetized in one or more RTP
packets, that completely resets the decoder to a known state.</t>
<t>Examples for "hard" single layer decoder refresh points are Intra
pictures in <xref target="H.261">H.261</xref>, <xref
target="H.263">H.263</xref>, <xref target="MPEG-1">MPEG-1</xref>, <xref
target="MPEG-2">MPEG-2</xref>, and <xref target="MPEG-4">MPEG-4</xref>;
Instantaneous Decoder Refresh (IDR) pictures in <xref
target="H.264">H.264</xref>, and <xref target="H.265">H.265</xref>; and
Keyframes in <xref target="RFC6386">VP8</xref> and <xref
target="I-D.grange-vp9-bitstream">VP9</xref>. "Gradual" decoder refresh
points may also be used; see for example <xref
target="H.264">H.264</xref>. While both "hard" and "gradual" decoder
refresh points are acceptable in the scope of this specification, in
most cases the user experience will benefit from using a "hard" decoder
refresh point.</t>
<t>A decoder refresh point also contains all header information above
the syntactical level of the picture layer (or equivalent, depending on
the video compression standard) that is conveyed in-band. In <xref
target="H.264"/>, for example, a decoder refresh point contains
parameter set Network Adaptation Layer (NAL) units that generate
parameter sets necessary for the decoding of the following slice/data
partition NAL units (and that are not conveyed out of band).</t>
<t>When a layered codec is in use, the above definition (and, in
particular, the requirement to COMPLETELY reset the decoder to a known
state) implies that the decoder refresh point includes hard or gradual
single layer decoder refresh points for all layers.</t>
</section>
<section title="Full Intra Request for Layered Codecs">
<t>
When a media receiver or middlebox has decided to send a FIR command
(based on the guidance provided in Section 4.3.1 of <xref
target="RFC5104"/>, it
MUST target the RTP stream that carries the base layer of the
layered bitstream, and this is done by setting the Feedback Control
Information (FCI, and in particular the SSRC field therein) to refer
to the SSRC of the forward RTP stream that carries the base layer. </t>
<t>When a Full Intra Request Command is received by the designated media
sender in the RTCP stream associated with any of the RTP streams in
which any layer of a layered bitstream are sent, the designated media
sender MUST send a <xref target="DRP">Decoder Refresh Point</xref> as
defined above at its earliest opportunity. The requirements related to
congestion control on the forward RTP streams as specified in sections
3.5.1 and 5. of <xref target="RFC5104"/> apply for the RTP streams both
in isolation and combined.</t>
<t>Note: the requirement to react to FIR commands associated with
enhancement layers is included for robustness and backward compatibility
reasons.</t>
</section>
<section title="Identifying the use of Layered Codecs (Informative)">
<t>The above modifications to RFC 5104 unambiguously define how to deal
with FIR when layered bitstreams are in use. However, it is surprisingly
difficult to identify this situation. In general, it is expected that
implementers know when layered coding (in its commonly understood sense:
with inter-layer prediction between pyramided-arranged layers) is in use
and when not, and can therefore implement the above updates to RFC 5104
correctly. However, there are use cases of the use of layered codecs
that may be viewed as somewhat exotic today but clearly are supported by
the video coding syntax, in which the above rules would lead to
suboptimal system behavior. Nothing would break, and there would not be
an interop failure, but the user experience may suffer through the
sending or receiving of Decoder Refresh Points at times or on parts of
the bitstream that are unnecessary from a user experience viewpoint.
Therefore, this informative section is included that provides the
current understanding of when a layered codec is in use and when
not.</t>
<t>The key observation made here is that the RTP payload format
negotiated for the RTP streams, in isolation, is not necessarily an
indicator for the use of layering. Some layered codecs (including
H.264/SVC) can form decodable bitstreams including only (one or more)
enhancement layers, without the base layer, effectively creating
simulcastable sub-bitstreams in a scalable bitstream that does not take
advantage of inter-layer prediction. In such a scenario, it is
potentially (though not necessarily) unnecessary--or even
counter-productive--to send a decoder refresh point on all RTP streams
using that payload format and SSRC.</t>
<t>One good indication of the likely use of layering with interlayer
prediction is when the various RTP streams are "bound" together on the
signaling level. In an SDP environment, this would be the case if they
are marked as being dependent from each other using <xref
target="RFC5888">The Session Description Protocol (SDP) Grouping
Framework</xref> and the layer dependency <xref
target="RFC5583">RFC 5583</xref>.</t>
</section>
<section title="Layered Codecs and non-FIR codec control messages (Informative)">
<t>Between them, <xref target="RFC4585">AVPF</xref> and <xref
target="RFC5104">Codec Control Messages</xref> define a total of seven
Payload-specific Feedback messages. For the FIR command message,
guidance has been provided above. In this section, some information is
provided with respect to the remaining six codec control messages.</t>
<section title="Picture Loss Indication (PLI)">
<t>PLI is defined in <xref target="RFC4585">section 6.3.1 of</xref>.
The prudent response to a PLI message received for an enhancement
layer is to "repair" (through whatever source-coding specific means)
that enhancement layer and all dependent enhancement layers, but not
the reference layer(s) used by the enhancement layer for which the PLI
was received. The encoder can figure out by itself what constitutes a
dependent enhancement layer and does not need help from the system
stack in doing so. Insofar, there is nothing that needs to be
specified herein.</t>
</section>
<section title="Slice Loss Indication (SLI)">
<t>SLI is defined in <xref target="RFC4585">section 6.3.2 of</xref>.
The authors' current understanding is that the prudent response to a
SLI message received for an enhancement layer is to "repair" (through
whatever source-coding specific means) the affected spatial area of
that enhancement layer and all dependent enhancement layers, but not
the reference layers used by the enhancement layer for which the SLI
was received. The encoder can figure out by itself what constitutes a
dependent enhancement layer and does not need help from the system
stack in doing so. Insofar, there is nothing that needs to be
specified herein. SLI has seen very little implementation and, as far
as it is known, none in conjunction with layered systems.</t>
</section>
<section anchor="RPSI"
title="Reference Picture Selection Indication (RPSI)">
<t>RPSI is defined in <xref target="RFC4585">section 6.3.3 of</xref>.
While a technical equivalent of RPSI has been in use with non-layered
systems for many years, no implementations are known in conjunction of
layered codecs. The authors' current understanding is that the
reception of an RPSI message on any layer indicating a missing
reference picture forces the encoder to appropriately handle that
missing reference picture in the layer indicated, and all dependent
layers.
Insofar, RPSI should work without further need for
specification language.</t>
</section>
<section title="Temporal-Spatial Trade-off Request and Notification (TSTR/TSTN)">
<t>TSTN/TSTR are defined in <xref target="RFC5104">section 4.3.2 and
4.3.3 of</xref>, respectively. The TSTR request allows to communicate
(typically user-interface-obtained) guidance of the preferred
trade-off between spatial quality and frame rate. A technical
equivalent of TSTN/TSTR has seen deployment for many years in
non-scalable systems.</t>
<t>The Temporal-Spatial Trade-off request and notification messages
include an SSRC target, which (similarly to FIR) may refer to an RTP
stream carrying a base layer, an enhancement layer, or multiple
layers. Therefore, the authors' current understanding is that the
semantics of the message applies to the layers present in the targeted
RTP stream.</t>
<t>It is noted that per-layer TSTR/TSTN is a mechanism that is, in
some ways, counterproductive in a system using layered codecs. Given a
sufficiently complex layered bitstream layout, a sending system has
flexibility in adjusting the spatio/temporal quality balance by adding
and removing temporal, spatial, or quality enhancement layers. At
present it is unclear whether an allowed (or even recommended) option
to the reception of a TSTR is to adjust the bit allocation within the
layer(s) present in the addressed RTP stream, or to adjust the
layering structure accordingly--which can involve more than just the
addressed RTP stream.</t>
<t>Until there is a sufficient critical mass of implementation
practice, it is probably prudent for an implementer not to assume
either of the two options (or any middleground that may exist between
the two), be liberal in accepting TSTR messages, perhaps responding in
TSTN indicating "no change," not sending TSTR messages except when
operating in SRST mode as defined in <xref target="RFC7656"/>, and
contribute to the IETF documentation of any implementation
requirements that make per-layer TSTR/TSTN useful.</t>
</section>
<section title="H.271 Video Back Channel Message (VBCM)">
<t>VBCM is defined in <xref target="RFC5104">section 4.3.4 of</xref>.
What was said above for <xref target="RPSI">RPSI</xref> applies here
as well.</t>
</section>
</section>
<section title="Acknowledgements">
<t>The authors want to thank Mo Zanaty for useful discussions.</t>
</section>
<section title="IANA Considerations">
<t>This memo includes no request to IANA.</t>
</section>
<section title="Security Considerations">
<t>The security considerations of <xref target="RFC4585">AVPF</xref> (as
updated by <xref target="RFC5506">Support for Reduced-Size Real-Time
Transport Control Protocol (RTCP): Opportunities and
Consequences</xref>) and <xref target="RFC5104">Codec Control
Messages</xref> apply. The clarified response to FIR does not require
any updates.</t>
</section>
</middle>
<!-- *****BACK MATTER ***** -->
<back>
<references title="Normative References">
<?rfc include='reference.RFC.2119'?>
<?rfc include='reference.RFC.4585'?>
<?rfc include='reference.RFC.5104'?>
<?rfc include="reference.RFC.5506"?>
</references>
<references title="Informative References">
<?rfc include='reference.RFC.5888'?>
<?rfc include='reference.RFC.5583'?>
<?rfc include='reference.RFC.7656'?>
<?rfc include='reference.RFC.6386'?>
<!--
<?rfc include='reference.I-D.ietf-mmusic-sdp-simulcast'?>
<?rfc include='reference.I-D.ietf-mmusic-rid'?>
-->
<?rfc include='reference.I-D.grange-vp9-bitstream'?>
<reference anchor="H.261"
target="http://handle.itu.int/11.1002/1000/1088">
<front>
<title>ITU-T Rec. H.261: Video codec for audiovisual services at p x
64 kbit/s</title>
<author>
<organization>ITU-T</organization>
</author>
<date year="1993"/>
</front>
</reference>
<reference anchor="H.263"
target="http://handle.itu.int/11.1002/1000/7497">
<front>
<title>ITU-T Rec. H.263: Video coding for low bit rate
communication</title>
<author>
<organization>ITU-T</organization>
</author>
<date year="2005"/>
</front>
</reference>
<reference anchor="H.264"
target="http://handle.itu.int/11.1002/1000/12063">
<front>
<title>ITU-T Rec. H.264: Advanced video coding for generic
audiovisual services</title>
<author>
<organization>ITU-T</organization>
</author>
<date year="2014"/>
</front>
</reference>
<reference anchor="H.265"
target="http://handle.itu.int/11.1002/1000/12455">
<front>
<title>ITU-T Rec. H.265: High efficiency video coding</title>
<author>
<organization>ITU-T</organization>
</author>
<date year="2015"/>
</front>
</reference>
<reference anchor="MPEG-1">
<front>
<title>ISO/IEC 11172-2:1993 Information technology -- Coding of
moving pictures and associated audio for digital storage media at up
to about 1,5 Mbit/s -- Part 2: Video</title>
<author>
<organization>ISO/IEC</organization>
</author>
<date year="1993"/>
</front>
</reference>
<reference anchor="MPEG-2">
<front>
<title>ISO/IEC 13818-2:2013 Information technology -- Generic coding
of moving pictures and associated audio information -- Part 2:
Video</title>
<author>
<organization>ISO/IEC</organization>
</author>
<date year="2013"/>
</front>
</reference>
<reference anchor="MPEG-4">
<front>
<title>ISO/IEC 14496-2:2004 Information technology -- Coding of
audio-visual objects -- Part 2: Visual</title>
<author>
<organization>ISO/IEC</organization>
</author>
<date year="2004"/>
</front>
</reference>
</references>
<section title="Change Log">
<t>NOTE TO RFC EDITOR: Please remove this section prior to
publication.</t>
<t>draft-wenger-avtext-avpf-ccm-layered-00-00: initial version</t>
<t>draft-ietf-avtext-avpf-ccm-layered-00: resubmit as avtext WG draft
per IETF95 and list confirmation by Rachel 4/25/2016</t>
<t>draft-ietf-avtext-avpf-ccm-layered-00: In section "Identifying the
use of Layered Codecs (Informative)", removed last sentence that could
be misread that the explicit signaling of simulcasting in conjunction
with payload formats supporting layered coding implies no layering.</t>
</section>
</back>
</rfc>
| PAFTECH AB 2003-2026 | 2026-04-24 04:25:38 |