One document matched: draft-alvestrand-idna-bidi-02.xml

<?xml version="1.0" encoding="US-ASCII"?>
<!DOCTYPE rfc SYSTEM "rfc2629.dtd">
<?rfc toc="yes"?>
<?rfc tocompact="yes"?>
<?rfc tocdepth="3"?>
<?rfc tocindent="yes"?>
<?rfc symrefs="yes"?>
<?rfc sortrefs="yes"?>
<?rfc comments="yes"?>
<?rfc inline="yes"?>
<?rfc compact="yes"?>
<?rfc subcompact="no"?>
<rfc category="std" docName="draft-alvestrand-idna-bidi-02" ipr="full3978">
  <front>
    <title abbrev="IDNA RTL fix">An IDNA problem in right-to-left
    scripts</title>

    <author fullname="Harald Tveit Alvestrand" initials="H. T." role="editor"
            surname="Alvestrand">
      <organization>Google</organization>

      <address>
        <postal>
          <street>Beddingen 10</street>

          <city>Trondheim</city>

          <region></region>

          <code>7014</code>

          <country>Norway</country>
        </postal>
	<email>harald@alvestrand.no</email>
      </address>
    </author>

    <author fullname="Cary Karp" initials="C." role="editor" surname="Karp">
      <organization>Swedish Museum of Natural History</organization>

      <address>
        <postal>
          <street>Frescativ. 40</street>

          <city>Stockholm</city>

          <region></region>

          <code>10405</code>

          <country>Sweden</country>
        </postal>

        <phone>+46 8 5195 4055</phone>

        <facsimile></facsimile>

        <email>ck@nrm.museum</email>

        <uri></uri>
      </address>
    </author>

    <date day="9" month="Jan" year="2008" />

    <abstract>
      <t>The use of right-to-left scripts in internationalized domain names
      has presented several challenges. This memo discusses some problems with
      these scripts, including one resulting from a constraint on the use of
      combining characters at the end of an RTL domain label, causing some
      words to be declared invalid as IDN labels, and proposes a means for
      ameliorating this problem.</t>
    </abstract>
  </front>

  <middle>
    <section title="Introduction and problem description">
      <t>The IDNA specification "Stringprep", <xref target="RFC3454"></xref>
      makes the following statement in its section 6 on the bidi algorithm,
      :<list style="empty">
          <t>3) If a string contains any RandALCat character, a RandALCat
          character MUST be the first character of the string, and a RandALCat
          character MUST be the last character of the string.</t>
        </list></t>

      <t>(A RandAlCat character is a character with unambiguously
      right-to-left directionality.)</t>

      <t>The reasoning behind this prohibition was to ensure that every
      component of a visually presented domain name has an unambiguously
      preferred direction. However, this makes certain words in languages
      written with right-to-left scripts invalid as IDN labels, and in at
      least one case means that all the words of an entire language are
      forbidden as IDN labels.</t>

      <t>This will be illustrated below with examples taken from the Dhivehi
      and Yiddish languages, as written with the Thaana and Hebrew scripts,
      respectively.</t>

      <t>The problem may be addressed by more carefully considering the bidi
      algorithm in Unicode Standard Annex #9 <xref target="UAX9"></xref> which
      states in section 3.3.3 W1: "Examine each non-spacing mark (NSM) in the
      level run, and change the type of the NSM to the type of the previous
      character." (See below for some terminology).</t>

      <t>Section 3 of UAX9 contains several instructions for determining the
      directionality of the characters in a string. Some of them (for instance
      those using explicit embedding) are irrelevant to IDNA because the
      corresponding codes are not permitted as IDNA input, so a slightly
      simplified version should be enough for IDNA purposes.</t>

      <t>A note on terminology:</t>

      <t>In this memo, we use "network order" to describe the sequence of
      characters as transmitted on the wire or stored in a file; the terms
      "first", "next" and "previous" are used to refer to the relationship of
      characters in network order.</t>

      <t>We use "display order" to talk about the sequence of characters as
      imaged on a display medium; the terms "left" and "right" are used to
      refer to the relationship of characters in display order.</t>
    </section>

    <section title="Detailed examples">
      <section title="Dhivehi">
        <t>Dhivehi, the official language of the Maldives, is written with the
        Thaana script. This displays some of the characteristics of Arabic
        script, including its directional properties, and the indication of
        vowels by the diacritical marking of consonantal base characters. This
        marking is obligatory, and both double vowels and syllable-final
        consonants are indicated by the marking of special unvoiced
        characters. Every Dhivehi word therefore ends with a combining
        mark.</t>

        <t>The word for "computer", which is romanized as "konpeetaru", is
        written with the following sequence of Unicode code points:</t>

        <t><list>
            <t>U+0786 THAANA LETTER KAAFU (AL)</t>

            <t>U+07AE THAANA OBOFILI (NSM)</t>

            <t>U+0782 THAANA LETTER NOONU (AL)</t>

            <t>U+07B0 THAANA SUKUN (NSM)</t>

            <t>U+0795 THAANA LETTER PAVIYANI (AL)</t>

            <t>U+07A9 THAANA LETTER EEBEEFILI (AL)</t>

            <t>U+0793 THAANA LETTER TAVIYANI (AL)</t>

            <t>U+07A6 THAANA ABAFILI (NSM)</t>

            <t>U+0783 THAANA LETTER RAA (AL)</t>

            <t>U+07AA THANAA UBIUFILI (NSM)</t>
          </list></t>

        <t>The directionality class of U+07AA in the Unicode database is NSM
        (non-spacing mark), which is not R or AL; a conformant implementation
        of the IDNA algorithm will say that "this is not in RandALCat", and
        refuse to encode the string.</t>
      </section>

      <section title="Yiddish">
        <t>Yiddish is one of several languages written with the Hebrew script
        (others include Hebrew and Ladino). This is basically a consonantal
        alphabet but Yiddish is written using an extended form that is fully
        vocalic. The vowels are indicated in several ways, of which one is by
        repurposing letters that are consonants in Hebrew. Other letters are
        used both as vowels and consonants, with combining marks used to
        differentiate between them. Finally, some base characters can indicate
        several different vowels, which are also disambiguated by combining
        marks. Marked characters can appear in word-final position and may
        therefore also be needed at the end of labels. This is not an
        invariable attribute of all Yiddish strings and there is thus greater
        latitude here than there is with Dhivehi.</t>

        <t>The "YIVO Institute for Jewish Research" is widely known by the
        acronym of its Yiddish name. This organization maintains a primary
        reference standard for modern Standard Yiddish orthography, that is
        also commonly referred to by the same acronym (as the "YIVO Rules").
        YIVO is written with the Hebrew letters YOD YOD HIRIQ VAV VAV ALEF
        QAMATS, where HIRIQ and QAMATS are combining "points":</t>

        <t><list>
            <t>U+05D9 HEBREW LETTER YOD (R)</t>

            <t>U+05B4 HEBREW POINT HIRIQ (NSM)</t>

            <t>U+05D5 HEBREW LETTER VAV (R)</t>

            <t>U+05D0 HEBREW LETTER ALEF (R)</t>

            <t>U+05B8 HEBREW POINT QAMATS (NSM)</t>
          </list></t>

        <t>The directionality class of U+05B8 HEBREW POINT QAMATS in the
        Unicode database is NSM, which again causes the IDNA algorithm to
        reject the string. (It may also be noted that the requisite combined
        characters also exist in precomposed form at separate positions in the
        Unicode chart. However, Stringprep also rejects those codepoints, for
        reasons not discussed here.)</t>
      </section>

      <section title="Strings with numbers">
        <t>RFC 3454, in its insistence that the first or last character of a
        string be category R or AL, prohibited strings that contained
        right-to-left characters and numbers.</t>

        <t>Considering the string ALEF 5 (HEBREW LETTER ALEF + DIGIT FIVE), if
        we specify that UAX#9 is used to find the directionality of
        characters, this string will have a consistent direction (R). However,
        the string 5 ALEF, when embedded in an LTR context, will have the same
        display order, with a different direction assigned to the number 5.
        These two display strings are confusable, so we need a rule that
        permits only one of these in a domain name label.</t>
      </section>
    </section>

    <section title="An expanded justification for the bidi rule">
      <t>One issue with RFC 3454 was that it did not give an explicit
      justification for the bidi rule, thus it was hard to tell if a modified
      rule would continue to fulfil the purpose for which the RFC 3454 rule
      was written.</t>

      <t>This document proposes an explicit justification, for which we think
      it is possible to test whether or not the modified rule fulfils the
      justification.</t>

      <t>The justification proposed is this:<list style="symbols">
          <t>No two labels, when presented in visual order, should have the
          same sequence of characters without also having the same sequence of
          characters in network order. (This is the criterion that is explicit
          in RFC 3454).</t>

          <t>In a visual presentation of a string of labels, the characters of
          each label should remain grouped between the dots delimiting the
          label components.</t>

          <t>This property should hold true both when the string is embedded
          in a RTL context and when it's embedded in a LTR context.</t>

          <t>This property should hold true without adding extra formatting,
          for example bidi control characters, to the string.</t>
        </list>Several stronger statements were considered and rejected,
      because they seem to be impossible to fulfil within the constraints of
      the Unicode bidirectional algorithm. These include:</t>

      <t><list style="symbols">
          <t>The appearance of a label should be unaffected by its embedding
          context. This proved impossible even for ASCII labels; the label
          "123-456" will have a different display order in a RTL context than
          in a LTR context.</t>

          <t>The sequence of labels should be consistent with network order.
          This proved impossible - a domain name consisting of the labels (in
          network order) L1.R1.R2.L2 will be displayed as L1.R2.R1.L2 in an
          LTR context.</t>
        </list></t>
    </section>

    <section title="Modification to RFC 3454">
      <t>If the following modification is made to RFC 3454 section 6,
      paragraph 4, we believe that the usefulness of the specification for
      languages written with right-to-left scripts will be significantly
      improved:</t>

      <t>Old text:</t>

      <t><list style="empty">
          <t>[Unicode3.2] defines several bidirectional categories; each
          character has one bidirectional category assigned to it. For the
          purposes of the requirements below, an "RandALCat character" is a
          character that has Unicode bidirectional categories "R" or "AL"; an
          "LCat character" is a character that has Unicode bidirectional
          category "L".</t>

          <t></t>
        </list>New text:</t>

      <t><list style="empty">
          <t>[Unicode3.2] defines several bidirectional categories; each
          character has one bidirectional category assigned to it.</t>

          <t>For characters that have category "R", "AL" or "L", the category
          is fixed (UAX#9 defines them as having "strong" category); for
          characters in category EN, ES, ET, AN, CS, NSM, BN, B, S, WS and ON,
          the category is determined by applying the algorithm described in
          UAX#9 section 3.3 to the string.</t>

          <t>For the purposes of the requirements below, an "RandALCat
          character" is a character that, after this determination, has
          Unicode bidirectional categories "R" or "AL"; an "LCat character" is
          a character that has Unicode bidirectional category "L".</t>
        </list></t>

      <t>Note that Unicode 5.0 is the current version of Unicode. This fix
      refers to Unicode 3.2 only, to maintain consistency with the rest of RFC
      3454. Nothing here should affect the relationship between Unicode
      versions and IDNA.</t>

      <t>Also, as noted in the introduction, the Unicode UAX#9 algorithm is
      quite complex. For the purposes of IDNA, a simpler algorithm may be
      defind that yields the same result within the constraints of this
      context, but may be easier for people to implement consistently. Such an
      algorithm may be included in later versions of this memo.</t>

      <t></t>

      <section title="Alternative approach">
        <t>The editors are not entirely happy with the text above. We are
        considering, instead, a complete replacement for section 6 of RFC
        3454.</t>

        <t>A first draft of such a section is below.</t>

        <t>Conceptually, to verify suitability as a domain name label, one
        constructs the string consisting of the label preceded and followed by
        a full stop (U+002E), and executes the Unicode bidirectional algorithm
        twice, once with <sor> (start of run) and <eor> (end of
        run) having direction L, and once with them having direction R. (The
        full stop, being of bidi class CS, is used because it seems likely to
        show up any problems, and occurs next to labels a lot of the time.
        Other times, a label is adjacent to an @ sign, a space or another
        character.)</t>

        <t>The following conditions MUST be true in both resulting strings for
        the string to be acceptable:</t>

        <t><list style="symbols">
            <t>The leftmost and rightmost character of the resulting string in
            display order must be a full stop (U+002E)</t>

            <t>No non-spacing mark (NSM) can occur in the second position of
            the string (leftmost in L order, rightmost in R order); that is,
            no mark can be allowed to attach to the delimiting characters.</t>

            <t>The direction of the leftmost and rightmost characters in the
            string (the periods) must be either L or R</t>
          </list>Note that there is no requirement that the character sequence
        be the same in the two cases.</t>

        <t>All RTL strings permitted by RFC 3454 section 6 will pass this
        test. Strings that consist of such a string with NSM characters
        appended to it will also pass this test.</t>

        <t>[[NOTE: Not sure if the ALEF 5 vs 5 ALEF issue will be solved by
        this rule. Test needed.]]</t>

        <t>[[NOTE: do we need to require something for the sor=L, eor=R and
        sor=R, eor=L cases?]]</t>

        <t></t>
      </section>
    </section>

    <section title="Other issues in need of resolution">
      <t>This is not the only issue with right-to-left scripts. Retaining
      Yiddish for the purposes of further exemplification, its alphabet
      includes three digraphs that can be encoded both as consecutive
      instances of the two component characters, and as precomposed ligatures.
      One of these digraphs also requires additional combined marking. For
      example, the HEBREW LIGATURE YIDDISH DOUBLE VAV (U+05F0) is
      orthographically equivalent to, and typographically utterly confusable
      with, a sequence of two HEBREW LETTER VAV (U+05D5). However, the
      ligature has no canonical decomposition and is therefore preserved by
      the IDNA algorithm. These digraphs need to be enumerated and the one
      form either made invalid for input in the IDNA context, or normalized to
      the other.</t>

      <t>We believe that there is a clear likelihood of similar issues
      existing with other scripts and languages that are not currently used
      extensively with IDNs. Careful consideration of all the languages
      written in a given script, in consultation with all of the corresponding
      speech communities, is therefore needed before we can say with any
      degree of certainty that using that script for IDNs is
      unproblematic.</t>

      <t>Another set of issues concerns the proper display of IDNs with a
      mixture of LTR and RTL labels, or only RTL labels; it is not clear to
      these authors what the proper display order of the components of a
      domain name are if the directiion of the components (in network order)
      is, for instance, FirstRTL.SecondRTL.LTR - is it LTRtsriF.LTRdnoceS.LTR
      or LTRdnoceS.LTRtsrif.LTR? Again, this memo does not attempt to suggest
      a solution to this problem.</t>
    </section>

    <section title="Backwards compatibility considerations">
      <t>As with any change to an existing standard, it is important to
      consider what happens with existing implementations when the change is
      introduced. The following troublesome cases have been noted:</t>

      <t><list style="symbols">
          <t>Old program used to input the newly allowed string. If the old
          program checks the input against RFC 3454, the string will not be
          allowed, and that domain name will remain inaccessible.</t>

          <t>Old program is asked to display the newly allowed string, and
          checks it against RFC 3454 before displaying. The program will
          perform some kind of fallback, most likely displaying the Punycode
          form of the string.</t>

          <t>Old program tries to display the newly allowed string. If the old
          program has code for displaying the last character of a string that
          is different from the code used to display the characters in the
          middle of the string, display may be inconsistent and cause
          confusion.</t>
        </list></t>

      <t>One particular example of the last case is if a program chooses to
      examine the last character (in network order) of a string in order to
      determine its directionality, rather than its first; if it finds an NSM
      character and tries to display the string as if it was a left-to-right
      string, the resulting display may be interesting, but not useful.</t>

      <t>The editors believe that these cases will have less harmful impact in
      practice than continuing to deny the use of words from the languages for
      which these strings are necessary as IDN labels.</t>
    </section>

    <section anchor="IANA" title="IANA Considerations">
      <t>This document makes no request of IANA.</t>

      <t>Note to RFC Editor: this section may be removed on publication as an
      RFC.</t>
    </section>

    <section anchor="Security" title="Security Considerations">
      <t>This modification will allow some strings to be used in Stringprep
      contexts that are not allowed today. It is possible that differences in
      the interpretation of the specification between old and new
      implementations could pose a security risk, but it is difficult to
      envision any specific instantiation of this.</t>

      <t>Any rational attempt to compute, for instance, a hash over an
      identifier processed by stringprep would use network order for its
      computation, and thus be unaffected by the changes proposed here.</t>

      <t>While it is not believed to pose a problem, if display routines had
      been written with specific knowledge of the current Stringprep
      prohibitions, it is possible that the possible problems noted under
      "backwards compatibility" could cause new kinds of confusion.</t>
    </section>

    <section anchor="Acknowledgements" title="Acknowledgements">
      <t>While the listed editors held the pen, this document represents the
      joint work and conclusions of an ad hoc design team. In addition to the
      editors this consisted of, in alphabetic order, Tina Dam, Patrik
      Faltstrom, and John Klensin. Many further specific contributions and
      helpful comments were received from the people listed below, and others
      who have contributed to the development and use of the IDNA
      protocols.</t>

      <t>The team wishes in particular to thank Roozbeh Pournader for calling
      its attention to the issue with the Thaana script, and Paul Hoffmann for
      pointing out the need to be explicit about backwards compatibility
      considerations.</t>
    </section>
  </middle>

  <back>
    <section title="Change log">
      <t>This appendix is intended to be removed when this document
         is published as an RFC.</t>
      <section title="Changes from -00 to -01">
      <t>Suggested a possible new algorithm.</t>
      <t>Multiple smaller changes.</t>
      </section>
      <section title="Changes from -01 to -02">
      <t>Date of publication updated.</t>
      <t>Change log added.</t>
      </section>
    </section>
    <references>
      <reference anchor="UAX9">
        <front>
          <title>Unicode Standard Annex #9: The Bidirectional Algorithm,
          revision 15</title>

          <author fullname="Mark Davis">
            <organization>0</organization>
          </author>

          <date day="25" month="03" year="2005" />
        </front>
      </reference>

      <?rfc include="reference.RFC.3454"?>
    </references>
  </back>
</rfc>