One document matched: draft-ietf-avtext-mixer-to-client-audio-level-02.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type='text/xsl' href='rfc2629.xslt' ?>
<!DOCTYPE rfc SYSTEM "rfc2629.dtd" [
<!ENTITY hdrext SYSTEM
'reference.RFC.5285.xml'>
<!ENTITY red SYSTEM
'reference.RFC.2198.xml'>
<!ENTITY rfc2119 SYSTEM
'reference.RFC.2119.xml'>
<!ENTITY rtp SYSTEM
'reference.RFC.3550.xml'>
<!ENTITY slic SYSTEM
'reference.I-D.ietf-avtext-mixer-to-client-audio-level.xml'>
<!ENTITY srtp SYSTEM
'reference.RFC.3711.xml'>
<!ENTITY hdrcrypt SYSTEM
'reference.I-D.lennox-avt-srtp-encrypted-extension-headers.xml'>
<!ENTITY vbr-srtp SYSTEM
'reference.I-D.perkins-avt-srtp-vbr-audio.xml'>
]>
<rfc category='info' ipr='trust200902'
docName='draft-ietf-avtext-mixer-to-client-audio-level-02'>
<?rfc toc='yes' ?>
<?rfc symrefs='yes' ?>
<?rfc sortrefs='yes'?>
<?rfc iprnotified='no' ?>
<?rfc strict='yes' ?>
<?rfc compact='yes' ?>
<front>
<title abbrev='Mixer-to-Client Audio Level Indication'>
A Real-Time Transport Protocol (RTP) Header Extension for
Mixer-to-Client Audio Level Indication
</title>
<author initials='E.' surname='Ivov'
fullname='Emil Ivov' role="editor">
<organization abbrev='Jitsi'>Jitsi</organization>
<address>
<postal>
<street></street>
<city>Strasbourg</city>
<code>67000</code>
<country>France</country>
</postal>
<email>emcho@jitsi.org</email>
</address>
</author>
<author initials='E.' surname='Marocco'
fullname='Enrico Marocco' role="editor">
<organization>Telecom Italia</organization>
<address>
<postal>
<street>Via G. Reiss Romoli, 274</street>
<city>Turin</city>
<code>10148</code>
<country>Italy</country>
</postal>
<email>enrico.marocco@telecomitalia.it</email>
</address>
</author>
<author initials='J.' surname='Lennox' fullname='Jonathan Lennox'>
<organization>Vidyo, Inc.</organization>
<address>
<postal>
<street>433 Hackensack Avenue</street>
<street>Seventh Floor</street>
<city>Hackensack,</city>
<code>NJ 07601</code>
<country>US</country>
</postal>
<email>jonathan@vidyo.com</email>
</address>
</author>
<date />
<abstract>
<t>
This document describes a mechanism for RTP-level mixers in
audio conferences to deliver information about the audio level
of individual participants. Such audio level indicators are
transported in the same RTP packets as the audio data they
pertain to.
</t>
</abstract>
</front>
<middle>
<section title='Introduction'>
<t>
The Framework for Conferencing with the Session Initiation
Protocol (SIP) defined in <xref target="RFC4353">RFC 4353</xref>
presents an overall architecture for multi-party conferencing.
Among others, the framework borrows from
<xref target="RFC3550">RTP</xref>
and extends the concept of a mixer entity "responsible for
combining the media streams that make up a conference, and
generating one or more output streams that are delivered to
recipients". Every participant would hence receive, in a flat
single stream, media originating from all the others.
</t>
<t>
Using such centralized mixer-based architectures simplifies
support for conference calls on the client side since they would
hardly differ from one-to-one conversations. However, the
method also introduces a few limitations. The flat nature of
the streams that a mixer would output and send to participants
makes it difficult for users to identify the original source of
what they are hearing.
</t>
<t>
Mechanisms that allow the mixer to send to participants cues on
current speakers (e.g. the CSRC fields in
<xref target='RFC3550'>RTP</xref>) only work for speaking/silent
binary indications. There are, however, a number of use cases
where one would require more detailed information. Possible
examples include the presence of background
chat/noise/music/typing, someone breathing noisily in their
microphone, or other cases where identifying the source of the
disturbance would make it easy to remove it (e.g. by sending a
private IM to the concerned party asking them to mute their
microphone). A more advanced scenario could involve an intense
discussion between multiple participants that the user does not
personally know. Audio level information would help better
recognize the speakers by associating with them complex (but
still human readable) characteristics like loudness and speed
for example.
</t>
<t>
One way of presenting such information in a user friendly
manner would be for a conferencing client to attach audio level
indicators to the corresponding participant related components
in the user interface as displayed in
<xref target='figure-conference-ui' />.
</t>
<figure title="Displaying detailed speaker information to the
user by including audio level for every participant."
anchor="figure-conference-ui">
<artwork>
<![CDATA[
________________________
| |
| 00:42 | Weekly Call |
|________________________|
| |
| |
| Alice |====== | (S) |
| |
| Bob |= | |
| |
| Carol | | (M) |
| |
| Dave |=== | |
| |
|________________________|
]]>
</artwork>
<postamble>
</postamble>
</figure>
<t>
Implementing a user interface like the above requires analysis
of the media sent from other participants. In a conventional
audio conference this is only possible for the mixer since all
other conference participants are generally receiving a single,
flat audio stream and have therefore no immediate way of
determining individual audio levels.
</t>
<t>
This document specifies an RTP extension header that allows such
mixers to deliver audio level information to conference
participants by including it directly in the RTP packets
transporting the corresponding audio data.
</t>
</section>
<section title="Terminology">
<t>
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL
NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and
"OPTIONAL" in this document are to be interpreted as described
in <xref target="RFC2119">RFC 2119</xref>.
</t>
</section>
<section title='Protocol Operation'>
<t>
According to <xref target='RFC3550'>RFC 3550</xref> a mixer
is expected to include in outgoing RTP packets a list of
identifiers (CSRC IDs) indicating the sources that contributed
to the resulting stream. The presence of such CSRC IDs allows
RTP clients to determine, in a binary way, the active speaker(s)
in any given moment. RTCP also provides a basic mechanism to map
the CSRC IDs to user identities through the CNAME field. More
advanced mechanisms, may exist depending on the signaling
protocol used to establish and control a conference. In the case
of the <xref target="RFC3261">Session Initiation Protocol</xref>
for example, the <xref target="RFC4575"> Event Package for
Conference State</xref> defines a <src-id> tag which binds
CSRC IDs to media streams and SIP URIs.
</t>
<t>
This document describes an RTP header extension that allows
mixers to indicate the audio-level of every conference
participant (CSRC) in addition to simply indicating their
on/off status. This new header extension uses "General Mechanism
for RTP Header Extensions" described in <xref
target="RFC5285"/>.
</t>
<t>
Each instance of this header contains a list of one-octet
audio levels expressed in -dBov, with values from 0 to 127
representing 0 to -127 dBov(see <xref target='hdr-fmt'/> and
<xref target='level-enc'/>). <xref target="ri"/> provides a
reference implementation indicating one way of obtaining such
values from raw audio samples.
</t>
<t>
Every audio level value pertains to the CSRC identifier
located at the corresponding position in the CSRC list. In other
words, the first value would indicate the audio level of the
conference participant represented by the first CSRC identifier
in that packet and so forth. The number and order of these
values MUST therefore match the number and order of the CSRC
IDs present in the same packet.
</t>
<t>
When encoding audio level information, a mixer SHOULD include in
a packet information that corresponds to the audio data being
transported in that same packet. It is important that these
values follow the actual stream as closely as possible.
Therefore a mixer SHOULD also calculate the values after the
original contributing stream has undergone possible processing
such as level normalization, and noise reduction for example.
</t>
<t>
Note that in some cases a mixer may be sending an RTP audio
stream that only contains audio level information and no actual
audio. Updating a (web) interface conference module may be one
reason for this to happen.
</t>
<t>
It may sometimes happen that a conference involves more than a
single mixer. In such cases each of the mixers MAY choose to
relay the CSRC list and audio-level information they receive
from peer mixers (as long as the total CSRC count remains below
16). Given that the maximum audio level is not precisely defined
by this specification, it is likely that in such situations
average audio levels would be perceptibly different for the
participants located behind the different mixers.
</t>
</section>
<section title='Header Format' anchor='hdr-fmt'>
<t>
The audio level indicators are delivered to the receivers
in-band using the <xref target='RFC5285'>"General Mechanism for
RTP Header Extensions"</xref>. The payload of this extension
is an ordered sequence of 8-bit audio level indicators encoded
as per <xref target="level-enc"/>.
<figure title="Audio level indicators extension format"
anchor="figure-hdr-fmt">
<preamble>
</preamble>
<artwork>
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ID | len |0| level 1 |0| level 2 |0| level 3 ...
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
</artwork>
</figure>
The 4-bit len field is the number minus one of data bytes (i.e.
audio level values) transported in this header extension element
following the one-byte header. Therefore, the value zero in this
field indicates that one byte of data follows.
<xref target="RFC3550">RFC 3550</xref> only allows RTP
packets to carry a maximum of 15 CSRC IDs. Given that audio
levels directly refer to CSRC IDs, implementations MUST NOT
include more than 15 audio level values. The maximum value
allowed in the len field is therefore 14.
</t>
<t>
Note that use of the two-byte header defined in
<xref target='RFC5285'> RFC 5285</xref> follows the same rules
the only change being the length of the ID and len fields.
</t>
</section>
<section title='Audio level encoding' anchor='level-enc'>
<t>
The audio level header extension only carries the level of
the audio in the RTP payload of the packet it is associated
with. This information is carried in an RTP header extension
element as defined by <xref target='RFC5285' />.
</t>
<t>
The audio level is defined in the same manner as is audio noise
level in the <xref target="RFC3389">RTP Payload Comfort
Noise specification</xref>. The overall magnitude of the noise
level is encoded into the first byte of the payload,
with spectral information about the noise in subsequent bytes.
This specification's audio level parameter is defined so as to be
identical to the comfort noise payload's noise-level byte.
</t>
<t>
The magnitude of the audio level is packed into the seven least
significant bits of the single byte of the header extension,
shown in <xref target='figure-level-enc'/>. The least significant
bit of the audio level magnitude is packed into the least
significant bit of the byte. The most significant bit of the
byte is unused and always set to 0 as shown below in
<xref target="figure-level-enc"/>.
<figure title="Audio Level Encoding" anchor="figure-level-enc">
<artwork>
<![CDATA[
0 1 2 3 4 5 6 7
+-+-+-+-+-+-+-+-+
|0| level |
+-+-+-+-+-+-+-+-+
]]>
</artwork>
<postamble>
</postamble>
</figure>
</t>
<t>
The two-byte header defined in <xref target='RFC5285'>RFC
5285</xref> may also be used.</t>
<t>
The audio level is expressed in -dBov, with values from 0 to 127
representing 0 to -127 dBov. dBov is the level, in decibels,
relative to the overload point of the system, i.e. the
maximum-amplitude signal that can be handled by the system
without clipping.(Note: Representation relative to the overload
point of a system is particularly useful for digital
implementations, since one does not need to know the relative
calibration of the analog circuitry.) For example, in the case
of <xref target='ITU.G.711'>u-law (audio/pcmu) audio</xref>,
the 0 dBov reference would be a square wave with values
+/- 8031. (This translates to 6.18 dBm0, relative to u-law's
dBm0 definition in Table 6 of G.711.)
</t>
<t>
The audio level for digital silence, for example for a muted
audio source, MAY be represented as 127 (-127 dBov),
regardless of the dynamic range of the encoded audio format.
</t>
<t>
Implementations MAY choose to measure audio levels prior to
encoding them in the payload carried in the RTP payload,
e.g. on raw linear PCM input.
</t>
<t>
The audio level header extension only carries the level of
the audio in the RTP payload of the packet it is associated
with, with no long-term averaging or smoothing applied.
</t>
<t>
To simplify implementation of the encoding procedures described
here, this specification provides a sample Java
<xref target="ri">implementation</xref> of an audio level
calculator that helps obtain such values from raw linear
PCM audio samples.
</t>
</section>
<section title='Signaling Information' anchor='sig-info'>
<t>
The URI for declaring the audio level header extension in an SDP
extmap attribute and mapping it to a local extension header
identifier is "urn:ietf:params:rtp-hdrext:csrc-audio-level".
There is no additional setup information needed for this
extension (i.e. no extensionattributes).
</t>
<t>
An example attribute line in the SDP, for a conference might be:
</t>
<figure>
<artwork>
a=extmap:7 urn:ietf:params:rtp-hdrext:csrc-audio-level
</artwork>
</figure>
<t>
The above mapping will most often be provided per media stream
(in the media-level section(s) of SDP, i.e., after an "m=" line)
or globally if there is more than one stream containing audio
level indicators in a session.
</t>
<t>
Presence of the above attribute in the SDP description of a
media stream indicates that some or all RTP packets in that
stream would contain the audio level information RTP extension
header.
</t>
<t>
Conferencing clients that support audio level indicators and
have no mixing capabilities would not be able to content for
this audio level extension and would hence have to always
include the direction parameter in the "extmap" attribute
with a value of "recvonly". Conference focus entities with
mixing capabilities can omit the direction or set it to
"sendrecv" in SDP offers. Such entities would need to
set it to "sendonly" in SDP answers to offers with a
"recvonly" parameter and to "sendrecv" when answering other
"sendrecv" offers.
</t>
<t>
This speicification does not define use of the audio level
extensions in video streams. Therefore, the extension defined
in this document SHOULD NOT be advertised in anything but
audio streams.
</t>
<t>
The following <xref target='client-focus'/> and <xref
target='focus-focus'/> show two example offer/answer exchanges
between a conferencing client and a focus, and between two
conference focus entities.
</t>
<figure anchor="client-focus">
<artwork>
v=0
o=alice 2890844526 2890844526 IN IP6 host.example.com
c=IN IP6 host.example.com
t=0 0
m=audio 49170 RTP/AVP 0 4
a=rtpmap:0 PCMU/8000
a=rtpmap:4 G723/8000
a=extmap:1/recvonly urn:ietf:params:rtp-hdrext:csrc-audio-level
v=0
i=A Seminar on the session description protocol
o=conf-focus 2890844730 2890844730 IN IP6 focus.example.net
c=IN IP6 focus.example.net
t=0 0
m=audio 52543 RTP/AVP 0
a=rtpmap:0 PCMU/8000
a=extmap:1/sendonly urn:ietf:params:rtp-hdrext:csrc-audio-level
</artwork>
<postamble>
A client-initiated example SDP offer/answer exchange
negotiating an audio stream with one-way flow of of audio
level information.
</postamble>
</figure>
<figure anchor="focus-focus">
<artwork>
v=0
i=Un seminaire sur le protocole de description des sessions
o=fr-focus 2890844730 2890844730 IN IP6 focus.fr.example.net
c=IN IP6 focus.fr.example.net
t=0 0
m=audio 49170 RTP/AVP 0
a=rtpmap:0 PCMU/8000
a=extmap:1/sendrecv urn:ietf:params:rtp-hdrext:csrc-audio-level
v=0
i=A Seminar on the session description protocol
o=us-focus 2890844526 2890844526 IN IP6 focus.us.example.net
c=IN IP6 focus.us.example.net
t=0 0
m=audio 52543 RTP/AVP 0
a=rtpmap:0 PCMU/8000
a=extmap:1/sendrecv urn:ietf:params:rtp-hdrext:csrc-audio-level
</artwork>
<postamble>
An example SDP offer/answer exchange between two conference
focus entities with mixing capabilities negotiating an audio
stream with bidirectional flow of audio level information.
</postamble>
</figure>
</section>
<section title='Security Considerations'>
<t>
<list style='numbers'>
<t>
This document defines a means of attributing audio level
to a particular participant in a conference. An attacker may
try to modify the content of RTP packets in a way that would
make audio activity from one participant appear as coming
from another.
</t>
<t>
Furthermore, the fact that audio level values would not be
protected even in an SRTP session might be of concern in
some cases where the activity of a particular participant
in a conference is confidential.
</t>
<t>
Both of the above are concerns that stem from the design of
the RTP protocol itself and they would probably also apply
when using CSRC identifiers the way they were specified in
<xref target="RFC3550">RFC 3550</xref>. It is therefore
important that according to the needs of a particular
scenario, implementors and deployers consider use of
<xref target='I-D.lennox-avtcore-srtp-encrypted-header-ext'>
header extension encryption</xref> or a lower
level security and authentication mechanism.
</t>
</list>
</t>
</section>
<section title='IANA Considerations'>
<t>
This document defines a new extension URI that, if approved,
would need to be added to the RTP Compact Header Extensions
sub-registry of the Real-Time Transport Protocol (RTP)
Parameters registry, according to the following data:
</t>
<figure>
<artwork>
Extension URI: urn:ietf:params:rtp-hdrext:csrc-audio-level
Description: Mixer-to-client audio level indicators
Contact: emcho@jitsi.org
Reference: RFC XXXX
</artwork>
</figure>
<t>Note to the RFC-Editor: please replace "RFC XXXX" by
the number of this RFC.
</t>
</section>
<section title="Acknowledgments">
<t>
Lyubomir Marinov contributed level measurement and rendering
code.
</t>
<t>
Roni Even, Keith Drage, Ingemar Johansson, Michael Ramalho
and several others provided helpful feedback over the
dispatch mailing list.
</t>
<t>
Jitsi's participation in this specification is funded by
the NLnet Foundation.
</t>
</section>
<section title='Changes From Earlier Versions'>
<t>
Note to the RFC-Editor: please remove this section prior to
publication as an RFC.
</t>
<section title='Changes From Draft -01'>
<t>
<list style='symbols'>
<t>
Removed code related the AudioLevelRenderer from
"APPENDIX A. Reference Implementation" as it was
considered an implementation matter by the working
group.
</t>
<t>
Modified the AudioLevelCalculator in "APPENDIX A.
Reference Implementation" to take overload as a
parameter.
</t>
<t>
Clarified non-use of audio levels in video streams
</t>
<t>
Closed the P.56 open issue. It was agreed on IETF 80
that P.56 is mostly about speech levels and the levels
transported by the extension defined here should also
be able to serve as an indication for noise.
</t>
<t>
The Open Issues section has been removed as all
issues that were in there are now resolved or clarified.
</t>
<t>Editorial changes for consistency with <xref
target="I-D.ietf-avtext-client-to-mixer-audio-level"/>.
</t>
</list>
</t>
</section>
<section title='Changes From Draft -00'>
<t>
<list style='symbols'>
<t>Added code for sound pressure calculation and measurement
in "APPENDIX A. Reference Implementation".</t>
<t>Changed affiliation for Emil Ivov.</t>
<t>Removed "Appendix: Design choices".</t>
</list>
</t>
</section>
</section>
</middle>
<back>
<references title='Normative References'>
<?rfc include="reference.RFC.2119"?>
<?rfc include="reference.RFC.3550"?>
<?rfc include="reference.RFC.5285"?>
</references>
<references title='Informative References'>
<?rfc include="reference.I-D.ietf-avtext-client-to-mixer-audio-level"?>
<?rfc include="reference.I-D.lennox-avtcore-srtp-encrypted-header-ext"?>
<?rfc include="reference.RFC.3261"?>
<?rfc include="reference.RFC.4353"?>
<?rfc include="reference.RFC.4575"?>
<?rfc include="reference.RFC.3389"?>
<reference anchor="ITU.G.711">
<front>
<title>Pulse Code Modulation (PCM) of Voice Frequencies</title>
<author>
<organization>
International Telecommunications Union
</organization>
</author>
<date month="November" year="1988" />
</front>
<seriesInfo name="ITU-T" value="Recommendation G.711" />
</reference>
<reference anchor="ITU.P56.1993">
<front>
<title>Objective Measurement of Active Speech Level</title>
<author>
<organization>
International Telecommunications Union
</organization>
</author>
<date month="March" year="1988" />
</front>
<seriesInfo name="ITU-T" value="Recommendation P.56" />
</reference>
</references>
<section title='Reference Implementation' anchor='ri'>
<t>
This appendix contains Java code for a reference implementation of
the level calculation and rendering methods.The code is not normative
and by no means the only possible implementation. Its purpose is to
help implementors add audio level support to mixers and clients.
</t>
<t>
The Java code contains an AudioLevelCalculator class that
calculates the sound pressure level of a signal with specific
samples. It can be used in mixers to generate values suitable
for the level extension headers.
</t>
<t>
The implementation is provided in Java but does not rely on
any of the language specific and can be easily ported to
another.
</t>
<section title='AudioLevelCalculator.java'>
<figure>
<artwork>
<![CDATA[
/**
* Calculates the audio level of specific samples of a singal based on
* sound pressure level.
*/
public class AudioLevelCalculator
{
/**
* Calculates the sound pressure level of a signal with specific
* <tt>samples</tt>.
*
* @param samples the samples of the signal to calculate the sound
* pressure level of. The samples are specified as an <tt>int</tt>
* array starting at <tt>offset</tt>, extending <tt>length</tt>
* number of elements and each <tt>int</tt> element in the specified
* range representing a sample of the signal to calculate the sound
* pressure level of. Though a sample is provided in the form of an
* <tt>int</tt> value, the sample size in bits is determined by the
* caller via <tt>overload</tt>.
*
* @param offset the offset in <tt>samples</tt> at which the samples
* start
*
* @param length the length of the signal specified in
* <tt>samples<tt> starting at <tt>offset</tt>
*
* @param overload the overload (point) of <tt>signal</tt>.
* For example, <tt>overload</tt> may be {@link Byte#MAX_VALUE}
* for 8-bit signed samples or {@link Short#MAX_VALUE} for
* 16-bit signed samples.
*
* @return the sound pressure level of the specified signal
*/
public static int calculateSoundPressureLevel(
int[] samples, int offset, int length,
int overload)
{
/*
* Calcuate the root mean square of the signal i.e. the
* effective sound pressure.
*/
double rms = 0;
for (; offset < length; offset++)
{
double sample = samples[offset];
sample /= overload;
rms += sample * sample;
}
rms = (length == 0) ? 0 : Math.sqrt(rms / length);
/*
* The sound pressure level is a logarithmic measure of the
* effectivesound pressure of a sound relative to a reference
* value and is measured in decibels.
*/
double db;
/*
* The minimum sound pressure level which matches the maximum
* of the sound meter.
*/
final double MIN_SOUND_PRESSURE_LEVEL = 0;
/*
* The maximum sound pressure level which matches the maximum
* of the sound meter.
*/
final double MAX_SOUND_PRESSURE_LEVEL
= 127 /* HUMAN TINNITUS (RINGING IN THE EARS) BEGINS */;
if (rms > 0)
{
/*
* The commonly used "zero" reference sound pressure in air
* is 20 uPa RMS, which is usually considered the threshold
* of human hearing.
*/
final double REF_SOUND_PRESSURE = 0.00002;
db = 20 * Math.log10(rms / REF_SOUND_PRESSURE);
/*
* Ensure that the calculated level is within the minimum
* and maximum sound pressure level.
*/
if (db < MIN_SOUND_PRESSURE_LEVEL)
db = MIN_SOUND_PRESSURE_LEVEL;
else if (db > MAX_SOUND_PRESSURE_LEVEL)
db = MAX_SOUND_PRESSURE_LEVEL;
}
else
{
db = MIN_SOUND_PRESSURE_LEVEL;
}
return (int) db;
}
}
]]>
</artwork>
<postamble>
AudioLevelCalculator.java
</postamble>
</figure>
</section>
</section>
</back>
</rfc>
| PAFTECH AB 2003-2026 | 2026-04-24 04:27:44 |