The Unicode Codepoints and IDN
draft-faltstrom-idnabis-tables-01.txt

Status of this Memo

By submitting this Internet-Draft, each author represents that any applicable patent or other IPR claims of which he or she is aware have been or will be disclosed, and any of which he or she becomes aware will be disclosed, in accordance with Section 6 of BCP 79.

Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet-Drafts.

Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as “work in progress.”

The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt.

The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html.

This Internet-Draft will expire on April 26, 2007.

Copyright Notice

Copyright © The Internet Society (2006).

Abstract

This document specifies the codepoints in the Unicode tables that are safe for use in the standards for Internationalized Domain Names, IDN.

Table of Contents

1.  Introduction
2.  Classes of Codepoints in Unicode
    2.1.  Classes of Code Points in Unicode 5.0
    2.2.  Problematic classes
        2.2.1.  Lu (Input)
        2.2.2.  Lt (Input)
        2.2.3.  Lo (Maybe)
        2.2.4.  Mn (Possibly not)
        2.2.5.  Mc (Maybe)
        2.2.6.  Me (Possibly not)
        2.2.7.  Nd (Maybe)
        2.2.8.  Nl (Possibly not)
        2.2.9.  Pd (Exclude)
        2.2.10.  Po (Exclude)
3.  Codepoint blocks in Unicode
    3.1.  Unicode Blocks
    3.2.  Notes
        3.2.1.  Note [1]
        3.2.2.  Note [2]
        3.2.3.  Note [3]
        3.2.4.  Note [4]
        3.2.5.  Note [5]
        3.2.6.  Note [6]
        3.2.7.  Note [7]
        3.2.8.  Note [8]
        3.2.9.  Note [9]
        3.2.10.  Note [10]
        3.2.11.  Note [11]
        3.2.12.  Note [12]
4.  Individual codepoints in Unicode
    4.1.  0000-007F Basic Latin
    4.2.  0080-00FF Latin-1 Supplement
    4.3.  0100-017F Latin Extended-A
    4.4.  0180-024F Latin Extended-B
    4.5.  0250-02AF IPA Extensions
    4.6.  02B0-02FF Spacing Modifier Letters
    4.7.  0300-036F Combining Diacritical Marks
    4.8.  0370-03FF Greek and Coptic
    4.9.  0400-04FF Cyrillic
    4.10.  0530-058F Armenian
    4.11.  0590-05FF Hebrew
    4.12.  0600-06FF Arabic
    4.13.  0700-074F Syriac
    4.14.  0750-077F Arabic supplement
    4.15.  0780-07BF Thaana
    4.16.  07C0-07FF NKo
    4.17.  0900-097F Devanagari
    4.18.  0980-09FF Bengali
    4.19.  0A00-0A7F Gurmukhi
    4.20.  0A80-0AFF Gujarati
    4.21.  0B00-0B7F Oriya
    4.22.  0B80-0BFF Tamil
    4.23.  0C00-0C7F Telugu
    4.24.  0C80-0CFF Kannada
    4.25.  0D00-0D7F Malayalam
    4.26.  0D80-0DFF Sinhala
    4.27.  0E00-0E7F Thai
    4.28.  0E80-0EFF Lao
    4.29.  0F00-0FFF Tibetan
5.  Outstanding Issues
6.  IANA Considerations
7.  Security Considerations
8.  Contributors
9.  Acknowledgements
10.  References
    10.1.  Normative References
    10.2.  Informative References
§  Author's Address
§  Intellectual Property and Copyright Statements

1.  Introduction

RFC 4690 (Klensin, J., Faltstrom, P., and Karp, C., “Review and Recommendations for Internationalized Domain Names (IDNs),” September 2006.) [RFC4690] describes an inclusion based approach for selecting the codepoints from The Unicode Standard (The Unicode Consortium, “The Unicode Standard, Version 5.0,” 2007.) [Unicode5] that should be included in the list of codepoints that may be used in Internationalized Domain Names.

Specifically, RFC 4690 (Klensin, J., Faltstrom, P., and Karp, C., “Review and Recommendations for Internationalized Domain Names (IDNs),” September 2006.) [RFC4690] says the following:

The IAB has concluded that there is a consensus within the broader community that lists of code points should be specified by the use of an inclusion-based mechanism (i.e., identifying the characters that are permitted), rather than by excluding a small number of characters from the total Unicode set as Stringprep (Hoffman, P. and M. Blanchet, “Preparation of Internationalized Strings ("stringprep"),” December 2002.) [RFC3454] and Nameprep (Hoffman, P. and M. Blanchet, “Nameprep: A Stringprep Profile for Internationalized Domain Names (IDN),” March 2003.) [RFC3491] do today. That conclusion should be reviewed by the IETF community and action taken as appropriate.

This document reviews the collections of codepoints in Unicode from two perspectives, those of character classes and those associated with individual characters and code blocks, in order to identify groups of characters that should clearly be included in IDNs, those that should clearly not be included, and those that still raise open issues, sometimes because there are complex trade offs involved and sometimes just due to lack of sufficient information. It is based on Unicode 5.0, rather than the earlier Unicode 3.2, in order to take advantage of the expanded character repertoire and better definitions in the newer version.

This document is part of a series that, together, constitute a preliminary proposal for updating the IDNA standards to resolve issues uncovered in recent years, cover a broader range of scripts, and provide for migration to newer versions of Unicode. See [IDNA‑issues] (Klensin, J., Ed., “Proposed Issues and Changes for IDNA - An Overview,” October 2006.) for a broader discussion.

2.  Classes of Codepoints in Unicode

The Unicode Standard (The Unicode Consortium, “The Unicode Standard, Version 5.0,” 2007.) [Unicode5] classifies the codepoints in a number of different classes. This table list the classes, and note in the "Comments" column whether characters of that class are appropriate to include or not. Classes not identified in this chart, such as "Cc" (control characters) are not permitted in IDNs.

2.1.  Classes of Code Points in Unicode 5.0

2.2.  Problematic classes

2.2.1.  Lu (Input)

The Class "Lu" is for Letter, Uppercase. As the DNS is case insensitive, all uppercase letters are to be case folded to lower case. This implies codepoints of class "Lu" will never exist when looking up a domain name, in a zone file or similar. In order to preserve as much symmetry as possible with ASCII domain names and the original DNS rules [RFC1035] (Mockapetris, P., “Domain names - implementation and specification,” November 1987.) should be allowed in Input (before case folding) according to the algorithm described in IDNAbis (Klensin, J., “Proposed Issues and Changes for IDNA - An Overview,” October 2006.) [idnabis].

2.2.2.  Lt (Input)

The Class "Lt" is for Letter, Titlecase. As with class "Lu", codepoints from this class should only be available in Input.

2.2.3.  Lo (Maybe)

The Class "Lo" is for Letter, Other. This class include codepoints that in general should be allowed to be used in IDN.

2.2.4.  Mn (Possibly not)

The Class "Mn" is for Mark, Nonspacing. This class includes codepoints that in general should not be allowed to be used in IDN, but some scripts do have codepoints in this class that should be allowed if possible since they are critical for writing the language.

2.2.5.  Mc (Maybe)

The Class "Mc" is for Mark, Spacing Combining. This class include codepoints that in general should not be allowed to use in IDN, but some scripts do have codepoints in this class that should be carefully considered as exceptions.

2.2.6.  Me (Possibly not)

The Class "Me" is for Mark, Enclosing.

2.2.7.  Nd (Maybe)

The Class "Nd" is for Number, Decimal Digit. This class includes codepoints that in general should be allowed for use in IDNs.

2.2.8.  Nl (Possibly not)

The Class "Nl" is for Number, Letter. This class include codepoints that in general should not be allowed to be used in IDN.

2.2.9.  Pd (Exclude)

The Class "Pd" is for Punctuation, Dash. This class include codepoints that in general should not be allowed to be used in IDNs, but the class include the '-' that is allowed by the basic DNS specifications and that cannot now be disallowed.

2.2.10.  Po (Exclude)

The Class "Po" is for Punctuation, Other. This class include codepoints that in general should not be allowed to be used in IDN.

3.  Codepoint blocks in Unicode

The following is a table that identifies the classes of codepoints that exist within each block of codepoints in The Unicode Standard. It also notes whether that block of codepoints is to be included or not in IDN.

3.1.  Unicode Blocks

3.2.  Notes

3.2.1.  Note [1]

A small number of IPA Extensions are used as extended Latin characters in some languages, and hence need to be segregated from this table and, if possible, made available in IDNs.

3.2.2.  Note [2]

The Unicode character class Mn (Mark, Nonspacing) does not distinguish between elements that are only meaningful in right-to-left scripts and elements that are only meaningful in left-to-right scripts. This is not adequate given other issues, such as those discussed in [IDNA‑bidi] (Alvestrand, H., Ed. and C. Karp, “An IDNA problem in right-to-left scripts,” Oct 2006.).

3.2.3.  Note [3]

Resolution of Note [2] is needed for adequate treatment. Significant cases have been identified in Dhivehi and Yiddish [IDNA‑bidi] (Alvestrand, H., Ed. and C. Karp, “An IDNA problem in right-to-left scripts,” Oct 2006.) are believed to exist in other languages written with right-to-left scripts that result in rejection of legitimate constructs with the current version of Stringprep. A major philosophical conflict arises here between combining and precomposed characters.

3.2.4.  Note [4]

Shaping issues require special attention for correct rendering and interpretation of characters. Precomposition is an issue as are some presentation forms that are defined in Unicode.

3.2.5.  Note [5]

These characters raise the issues noted in Note [4] and some additional issues with combining characters.

3.2.6.  Note [6]

Verticality has not been fully considered in the DNS context.

3.2.7.  Note [7]

Subject of significant controversy at present. This document recommends against adding to DNS list until the controversy is resolved.

3.2.8.  Note [8]

Requires advice of expert community. It is not clear whether some issues with Unicode definitions for these characters and the associated script have been fully resolved.

3.2.9.  Note [9]

Important language community, but characters have significant potential for confusion.

3.2.10.  Note [10]

Script not believed to be in current use, even by research communities relevant to IDN.

3.2.11.  Note [11]

Requires more information and advice from expert community.

3.2.12.  Note [12]

CJK (and related) radicals, compatibility, composing and Chinese phonetic characters are believed to be inappropriate on the basis of JET and CDNC work (see RFC 4713 (Lee, X., Mao, W., Chen, E., Hsu, N., and J. Klensin, “Registration and Administration Recommendations for Chinese Domain Names,” October 2006.) [RFC4713]).

4.  Individual codepoints in Unicode

If, instead, we examine the individual codepoints of Unicode Codepoints between U+0000 and U+0FFF, retaining the decisions made based on the classes above, we get the following result.

4.1.  0000-007F Basic Latin

4.2.  0080-00FF Latin-1 Supplement

4.3.  0100-017F Latin Extended-A

4.4.  0180-024F Latin Extended-B

4.5.  0250-02AF IPA Extensions

4.6.  02B0-02FF Spacing Modifier Letters

4.7.  0300-036F Combining Diacritical Marks