One document matched: draft-alvestrand-idna-bidi-03.xml
<?xml version="1.0" encoding="US-ASCII"?>
<!DOCTYPE rfc SYSTEM "rfc2629.dtd">
<?rfc toc="yes"?>
<?rfc tocompact="yes"?>
<?rfc tocdepth="3"?>
<?rfc tocindent="yes"?>
<?rfc symrefs="yes"?>
<?rfc sortrefs="yes"?>
<?rfc comments="yes"?>
<?rfc inline="yes"?>
<?rfc compact="yes"?>
<?rfc subcompact="no"?>
<rfc category="std" docName="draft-alvestrand-idna-bidi-03" ipr="full3978">
<front>
<title abbrev="IDNA RTL fix">An updated IDNA criterion for right-to-left
scripts</title>
<author fullname="Harald Tveit Alvestrand" initials="H. T." role="editor"
surname="Alvestrand">
<organization>Google</organization>
<address>
<postal>
<street>Beddingen 10</street>
<city>Trondheim</city>
<region></region>
<code>7014</code>
<country>Norway</country>
</postal>
<email>harald@alvestrand.no</email>
</address>
</author>
<author fullname="Cary Karp" initials="C." role="editor" surname="Karp">
<organization>Swedish Museum of Natural History</organization>
<address>
<postal>
<street>Frescativ. 40</street>
<city>Stockholm</city>
<region></region>
<code>10405</code>
<country>Sweden</country>
</postal>
<phone>+46 8 5195 4055</phone>
<facsimile></facsimile>
<email>ck@nrm.museum</email>
<uri></uri>
</address>
</author>
<date day="27" month="Jan" year="2008" />
<abstract>
<t>The use of right-to-left scripts in internationalized domain names
has presented several challenges. This memo discusses some problems with
these scripts, and some shortcomings in the 2003 IDNA BIDI criterion.
Based on this discussion, it proposes a new BIDI criterion for
IDNA labels.</t>
</abstract>
</front>
<middle>
<section title="Introduction and problem description">
<t>The IDNA specification "Stringprep", <xref target="RFC3454"></xref>
makes the following statement in its section 6 on the bidi algorithm,
:<list style="empty">
<t>3) If a string contains any RandALCat character, a RandALCat
character MUST be the first character of the string, and a RandALCat
character MUST be the last character of the string.</t>
</list></t>
<t>(A RandAlCat character is a character with unambiguously
right-to-left directionality.)</t>
<t>The reasoning behind this prohibition was to ensure that every
component of a displayed domain name has an unambiguously preferred
direction. However, this makes certain words in languages written with
right-to-left scripts invalid as IDN labels, and in at least one case
means that all the words of an entire language are forbidden as IDN
labels.</t>
<t>This will be illustrated below with examples taken from the Dhivehi
and Yiddish languages, as written with the Thaana and Hebrew scripts,
respectively.</t>
<t>In investigating this problem, it was realized that the RFC 3454
specification did not exactly specify what the requirement to be
fulfilled was, and therefore, it was impossible to tell whether a simple
relaxation of the rule would continue to fulfil the requirement. A
further investigation led to the conclusion that for one reasonable set
of requirements, IDNA2003's BIDI restriction did not fulfil the
requirements. This document therefore proposes replacing the RFC 3454
BIDI requirement in its entirety.</t>
<t>While the document proposes completely new text, most
reasonable labels that were allowed under the old criterion will
also be allowed under the new criterion, so the operational
impact of the rule change is limited.</t>
<t>A note on terminology:</t>
<t>In this memo, we use "network order" to describe the sequence of
characters as transmitted on the wire or stored in a file; the terms
"first", "next" and "previous" are used to refer to the relationship of
characters in network order.</t>
<t>We use "display order" to talk about the sequence of characters as
imaged on a display medium; the terms "left" and "right" are used to
refer to the relationship of characters in display order.</t>
</section>
<section title="Detailed examples">
<section title="Dhivehi">
<t>Dhivehi, the official language of the Maldives, is written with the
Thaana script. This displays some of the characteristics of Arabic
script, including its directional properties, and the indication of
vowels by the diacritical marking of consonantal base characters. This
marking is obligatory, and both double vowels and syllable-final
consonants are indicated by the marking of special unvoiced
characters. Every Dhivehi word therefore ends with a combining
mark.</t>
<t>The word for "computer", which is romanized as "konpeetaru", is
written with the following sequence of Unicode code points:</t>
<t><list>
<t>U+0786 THAANA LETTER KAAFU (AL)</t>
<t>U+07AE THAANA OBOFILI (NSM)</t>
<t>U+0782 THAANA LETTER NOONU (AL)</t>
<t>U+07B0 THAANA SUKUN (NSM)</t>
<t>U+0795 THAANA LETTER PAVIYANI (AL)</t>
<t>U+07A9 THAANA LETTER EEBEEFILI (AL)</t>
<t>U+0793 THAANA LETTER TAVIYANI (AL)</t>
<t>U+07A6 THAANA ABAFILI (NSM)</t>
<t>U+0783 THAANA LETTER RAA (AL)</t>
<t>U+07AA THANAA UBIUFILI (NSM)</t>
</list></t>
<t>The directionality class of U+07AA in the Unicode database is NSM
(non-spacing mark), which is not R or AL; a conformant implementation
of the IDNA algorithm will say that "this is not in RandALCat", and
refuse to encode the string.</t>
</section>
<section title="Yiddish">
<t>
Yiddish is one of several languages written with the Hebrew script
(others include Hebrew and Ladino). This is basically a consonantal
alphabet (also termed an "abjad") but Yiddish is written using an
extended form that is fully vocalic. The vowels are indicated in
several ways, of which one is by repurposing letters that are
consonants in Hebrew. Other letters are used both as vowels and
consonants, with combining marks, called "points", used to
differentiate between them. Finally, some base characters can indicate
several different vowels, which are also disambiguated by combining
marks. Pointed characters can appear in word-final position and may
therefore also be needed at the end of labels. This is not an
invariable attribute of a Yiddish string and there is thus greater
latitude here than there is with Dhivehi.
</t>
<t>
The organization now known as the "YIVO Institute for Jewish Research"
developed orthographic rules for modern Standard Yiddish during the
1930s on the basis of work conducted in several venues since earlier in
that century. These are given in, "The Standardized Yiddish
Orthography: Rules of Yiddish Spelling, 6th ed., YIVO Institute for
Jewish Research, New York, 1999, ISBN 0-914512-25-0", ("SYO") and
are taken as normatively descriptive of modern Standard Yiddish in any
context where that notion is deemed relevant. They have been applied
exclusively in all Yiddish dictionaries published since their
establishment, and are similarly dominant in academic and bibliographic
regards.
</t>
<t>
It therefore appears appropriate for this repertoire also to be
supported fully by IDNA. This presents no difficulty with characters
in initial and medial positions, but pointed characters are regularly
used in final position as well. All of the characters in the SYO
repertoire appear in both marked and unmarked form with one exception:
the HEBREW LETTER PE (U+05E4). The SYO only permits this with a HEBREW
POINT DAGESH (U+05BC), providing the Yiddish equivalent to the Latin
letter "p", or a HEBREW POINT RAFE (U+05BF), equivalent to the Latin
letter "f". There is, however, a separate unpointed allograph, the
HEBREW LETTER FINAL PE (U+05E3), for the latter character when it
appears in final position. The constraint on the use of the SYO
repertoire resulting from the proscription of combining marks at the
end of RTL strings thus reduces to nothing more, or less, than the
equivalent of saying that a string of Latin characters cannot end with
the letter "p". It must also be noted that the HEBREW LETTER PE with
HEBREW POINT DAGESH is characteristic of almost all traditional Yiddish
orthographies that predate (or remain in use in parallel to) the SYO,
being the first pointed character to appear in any of them.
</t>
<t>
A more general instantiation of the basic problem can be seen in the
representation of the YIVO acronym. This is written with the Hebrew
letters YOD YOD HIRIQ VAV VAV ALEF QAMATS, where HIRIQ and QAMATS are
combining points:
<list>
<t>U+05D9 HEBREW LETTER YOD (R)</t>
<t>U+05B4 HEBREW POINT HIRIQ (NSM)</t>
<t>U+05D5 HEBREW LETTER VAV (R)</t>
<t>U+05D0 HEBREW LETTER ALEF (R)</t>
<t>U+05B8 HEBREW POINT QAMATS (NSM)</t>
</list>
The directionality class of U+05B8 HEBREW POINT QAMATS in the
Unicode database is NSM, which again causes the IDNA2003 algorithm to
reject the string.
</t>
<t>
It may also be noted that all of the combined characters mentioned
above exist in precomposed form at separate positions in the Unicode
chart. However, by invoking Stringprep, the IDNA2003 algorithm also
rejects those codepoints, for reasons not discussed here.
</t>
</section>
<section title="Strings with numbers">
<t>RFC 3454, in its insistence that the first or last character of a
string be category R or AL, prohibited strings that contained
right-to-left characters and numbers at the end.</t>
<t>Considering the strings ALEF 5 (HEBREW LETTER ALEF + DIGIT
FIVE and 5 ALEF. Displayed in a LTR context, the first one
will be displayed from left to right as 5 ALEF (with the 5
being considered right-to-left because of the leading ALEF),
while 5 ALEF will be displayed in exactly the same order (5
taking the direction from context). Clearly, only one of
those should be permitted as a registered label.</t>
</section>
</section>
<section title="An expanded justification for the bidi rule">
<t>One issue with RFC 3454 was that it did not give an explicit
justification for the bidi rule, thus it was hard to tell if a modified
rule would continue to fulfil the purpose for which the RFC 3454 rule
was written.</t>
<t>This document proposes an explicit justification, by stating a set of
requirements for which we think it is possible to test whether or not
the modified rule fulfils the requirement.</t>
<t>All the text in this document assumes that text containing
the labels under consideration will be displayed using the
Unicode bidirectional algorithm <xref target="UAX9"></xref>.</t>
<t>The justification proposed is this:<list style="symbols">
<t>No two labels, when presented in display order, should have the
same sequence of characters without also having the same sequence of
characters in network order. (This is the criterion that is explicit
in RFC 3454).</t>
<t>In a display of a string of labels, the characters of each label
should remain grouped between the characters delimiting the label
components.</t>
<t>These properties should hold true both when the string is
embedded in a paragraph with LTR direction and when it's embedded in
a paragraph with RTL direction, as long as explicit directional
controls are not used within the same paragraph.</t>
</list>
</t>
<t>
Several stronger statements were considered and rejected,
because they seem to be impossible to fulfil within the constraints of
the Unicode bidirectional algorithm. These include:</t>
<t><list style="symbols">
<t>The appearance of a label should be unaffected by its embedding
context. This proved impossible even for ASCII labels; the label
"123-456" will have a different display order in a RTL context than
in a LTR context.</t>
<t>The sequence of labels should be consistent with network order.
This proved impossible - a domain name consisting of the labels (in
network order) L1.R1.R2.L2 will be displayed as L1.R2.R1.L2 in an
LTR context.</t>
<t>The "remain grouped" property should remain true when directional
controls (LRE, RLE, RLO, LRO, PDF) are used in the same paragraph
(outside of the labels). Because these controls affect presentation
order in non-obvious ways, by affecting the "sor" and "eor"
properties of the Unicode BIDI algorithm, the conditions above would
be very hard to satisfy for an useful set of strings if this was
true. As long as these controls have no influence over the
display of the domain name, no problem will be caused, but
the exact criterion for "will not influence" is hard to codify.</t>
<t>The "no two labels display the same" should hold true between LTR
paragraphs and RTL paragraphs. This was shown to be unsound, since
((EXAMPLE NEEDED HERE).</t>
<t>No two domain names should be displayed the same, even
under differing directionality. This was shown to be
unsound, since the domain name (network) ABC.abc will have display
order CBA.abc in an LTR context and abc.CBA in an RTL context,
while the domain name (network) abc.ABC will display as
abc.CBA in an LTR context and as CBA.abc in a RTL context.</t>
</list></t>
<t>For reference, here are the values that the Unicode BIDI
property can have:
<list style="symbols">
<t>L - Left-to-right - most letters in LTR scripts</t>
<t>R - Right-to-left - most letters in non-Arabic RTL scripts</t>
<t>AL - Arabic letters - most letters in the Arabic
script</t>
<t>EN - European Number (0-9)</t>
<t>ES - European Number Separator (+ and -)</t>
<t>ET - European Number Terminator (currency symbols, the
hash sign, the percent sign and so on)</t>
<t>AN - Arabic Number</t>
<t>CS - Common Number Separator (. , / : et al)</t>
<t>NSM - Nonspacking Mark - most combining accents</t>
<t>BN - Boundary Neutral - control characters</t>
<t>B - Paragraph Separator</t>
<t>S - Segment Separator</t>
<t>WS - Whitespace, including the SPACE character</t>
<t>ON - Other Neutrals, including @, &, parentheses, MIDDLE DOT</t>
<t>LRE, LRO, RLE, RLO, PDF - these are "directional control
characters", and are not used in IDNA labels.</t>
</list>
</t>
<t>The "remain grouped" property can be more formally stated as:<list
style="symbols">
<t>Let "Delimiterchars" be a set of characters with the Unicode
BIDI properties CS, WS, ON
<list style="symbols">
<t>ET, which commonly occurs next to domain names in practice,
is problematic: the context R CS L EN ET (for instance A.a1%)
makes the label L EN grow unstable.</t>
<t>ES commonly occurs in labels as HYPHEN-MINUS, but
could also be used as a delimiter (for instance, the
plus sign). It is left out here.</t>
</list></t>
<t>Let "Position" be the position of a character in a string (in
network order)</t>
<t>Let "Bidi position" be the position computed by the Unicode Bidi
algorithm</t>
</list>In the paragraph containing a string formed from the substrings
A B L C D, where A and D are (possibly zero-length) legal labels, and B
and C are single "Delimiterchars", the label L is a legal label if, for
all A, B, C and D, the bidi position of all characters in L is within
the range of positions for the characters of L in the string, for both
the LTR and RTL paragraph direction.</t>
<t>The "No two labels" property can be formally stated as:</t>
<t>If two labels L and L', embedded as for the test above, displayed in
a paragraph with the same directionality, are rearranged into the same
sequence of codepoints, neither L nor L' is a legal label.</t>
</section>
<section title="A replacement for the RFC 3454 criterion">
<t>A set of rules that satisfies the tests above is as follows. The main
bullets give the rule, subordinate bullets (if any) give justifications
or examples of things that break if this rule is not present. The term
"unstable" means that it fails to satisfy the "remain grouped" property
defined above.</t>
<t>NOTE: The "remain grouped" property has been tested
exhaustively up to 5-character strings with 0-3 character
surrounding labels. The "no two labels display the same" property has not
been tested.</t>
<t><list style="symbols">
<t>Only characters with the BIDI properties L, R, AL, EN, ES, BN, ON
and NSM are allowed.<list style="symbols">
<t>B, S and WS are excluded because they are separators or
spaces.</t>
<t>LRE, LRO, RLE, RLO, PDF are excluded because they are bidi
controls.</t>
<t>ET is excluded because the string L ET is unstable.</t>
<t>CS is excluded because the string L CS is unstable.</t>
</list></t>
<t>ES and ON are not allowed in the first position<list
style="symbols">
<t>ES R and ON R are both unstable.</t>
</list> </t>
<t>ES and ON, followed by zero or more NSM, is not allowed in the
last position<list style="symbols">
<t>L ON and L ES are both unstable.</t>
</list></t>
<t>If an L is present, no R, AL or AN may be present.</t>
<t>If an R, AL or AN is present, no L may be present.</t>
<t>If an EN is present, no AN may be present</t>
<t>If an AN is present, no EN may be present</t>
<t>If an AN is present, at least one R or AL must be present
<list style="symbols">
<t>if 1 or more AN are allowed alone, AL AN, when put next to
AN, is unstable.</t>
<t>The same thing happens with AL ES AN, AL ON AN and AL NSM</t>
<t>A BN has no influence on the Bidi algorithm, so doesn't
help.</t>
<t>EN is not permitted per the rule above.</t>
</list></t>
<t>The first character may not be an NSM</t>
<t>The first character may not be an EN (European Number) or an AN
(Arabic Number).<list style="symbols">
<t>If the character on both sides of a CS is an EN or an AN, the
labels turn unstable.</t>
<t>Some domain names where some of the labels use leading EN and
AN may be problem-free, but there's no way of verifying this
while looking at a single label in isolation.</t>
<t>NOTE: This is a restriction on ASCII labels when used
together with IDNA labels. This is a change from the existing
rules for ASCII labels.</t>
<t>We could achieve stability by barring numbers at the end of
labels, but this may be more disruptive in practice.</t>
</list></t>
</list></t>
</section>
<section title="Other issues in need of resolution">
<t>This document concerns itself only with the rules that are
needed when dealing with domain names with characters that have
differing Bidi properties, and considers characters only in
terms of their Bidi properties. All other issues with these
scripts have to be considered in other contexts.</t>
<t>Another set of issues concerns the proper display of IDNs with a
mixture of LTR and RTL labels, or only RTL labels.</t>
<t>It is unrealistic to expect that domain names will be written using
embedded formatting codes between their labels; thus, the display order
will be determined by the bidirectional algorithm. Thus, a
sequence (in network order) of R1.R2.ltr will be displayed
in the order 2R.1R.ltr in a LTR context, which might surprise
someone expecting to see labels displayed in hierarchical
order. Again, this memo does not attempt to suggest
a solution to this problem.</t>
<t></t>
</section>
<section title="Compatibility considerations">
<t></t>
<section title="Backwards compatibility considerations">
<t>As with any change to an existing standard, it is important to
consider what happens with existing implementations when the change is
introduced. The following troublesome cases have been noted:</t>
<t><list style="symbols">
<t>Old program used to input the newly allowed string. If the old
program checks the input against RFC 3454, the string will not be
allowed, and that domain name will remain inaccessible.</t>
<t>Old program is asked to display the newly allowed string, and
checks it against RFC 3454 before displaying. The program will
perform some kind of fallback, most likely displaying the Punycode
form of the string.</t>
<t>Old program tries to display the newly allowed string. If the
old program has code for displaying the last character of a string
that is different from the code used to display the characters in
the middle of the string, display may be inconsistent and cause
confusion.</t>
</list></t>
<t>One particular example of the last case is if a program chooses to
examine the last character (in network order) of a string in order to
determine its directionality, rather than its first; if it finds an
NSM character and tries to display the string as if it was a
left-to-right string, the resulting display may be interesting, but
not useful.</t>
<t>The editors believe that these cases will have less harmful impact
in practice than continuing to deny the use of words from the
languages for which these strings are necessary as IDN labels.</t>
</section>
<section title="Forward compatibiltiy considerations">
<t>This text is, intentionally, specified strictly in terms of the
Unicode BIDI properties. The determination that the condition is
sufficient to fulfil the criteria depends on the Unicode BIDI
algorithm; it is unlikely that drastic changes will be made to this
algorithm.</t>
<t>However, the determination of validity for any string depends on
the Unicode BIDI property values, which are not declared immutable by
the Unicode Consortium. Furthermore, the behaviour of the algorithm
for any given character is likely to be linguistically and culturally
sensitive, so that it's not unlikely that later versions of the
Unicode standard may change the bidi properties assigned to certain
Unicode characters.</t>
<t>This memo does not propose a solution for this problem.</t>
</section>
</section>
<section anchor="IANA" title="IANA Considerations">
<t>This document makes no request of IANA.</t>
<t>Note to RFC Editor: this section may be removed on publication as an
RFC.</t>
</section>
<section anchor="Security" title="Security Considerations">
<t>This modification will allow some strings to be used in Stringprep
contexts that are not allowed today. It is possible that differences in
the interpretation of the specification between old and new
implementations could pose a security risk, but it is difficult to
envision any specific instantiation of this.</t>
<t>Any rational attempt to compute, for instance, a hash over an
identifier processed by stringprep would use network order for its
computation, and thus be unaffected by the changes proposed here.</t>
<t>While it is not believed to pose a problem, if display routines had
been written with specific knowledge of the RFC 3454 Stringprep
prohibitions, it is possible that the possible problems noted under
"backwards compatibility" could cause new kinds of confusion.</t>
</section>
<section anchor="Acknowledgements" title="Acknowledgements">
<t>While the listed editors held the pen, this document represents the
joint work and conclusions of an ad hoc design team. In addition to the
editors this consisted of, in alphabetic order, Tina Dam, Patrik
Faltstrom, and John Klensin. Many further specific contributions and
helpful comments were received from the people listed below, and others
who have contributed to the development and use of the IDNA
protocols.</t>
<t>The team wishes in particular to thank Roozbeh Pournader for calling
its attention to the issue with the Thaana script, Paul Hoffmann for
pointing out the need to be explicit about backwards compatibility
considerations, and Ken Whistler for suggesting the basis of the
formalized "remain grouped" requirement.</t>
</section>
</middle>
<back>
<section title="Change log">
<t>This appendix is intended to be removed when this document is
published as an RFC.</t>
<section title="Changes from -00 to -01">
<t>Suggested a possible new algorithm.</t>
<t>Multiple smaller changes.</t>
</section>
<section title="Changes from -01 to -02">
<t>Date of publication updated.</t>
<t>Change log added.</t>
</section>
<section title="Changes from -02 to -03">
<t>Intro changed to reflect addresing the deeper issues with the Bidi
algorithm.</t>
<t>Gave formalized criteria for "valid strings", and documented the
new set of requirements for strings that satisfy the
criteria.</t>
<t>Removed most of section 5, "Other problems", and noted that
this memo focuses ONLY on issues that can be evaluated by
looking at the bidi properties of characters.</t>
</section>
</section>
<references>
<reference anchor="UAX9">
<front>
<title>Unicode Standard Annex #9: The Bidirectional Algorithm,
revision 15</title>
<author fullname="Mark Davis" initials="M." surname="Davis">
<organization>Unicode Consortium</organization>
</author>
<date day="25" month="03" year="2005" />
</front>
</reference>
<?rfc include="reference.RFC.3454"?>
</references>
</back>
</rfc>
| PAFTECH AB 2003-2024 | 2024-05-10 10:17:04 |