UTS #22: CharMapML Technical Reports Unicode Technical Standard #22 Unicode Character Mapping Markup Language (CharMapML) Version 5.0.1 Authors Mark Davis, Markus Scherer Date 2017-05-31 This Version http://www.unicode.org/reports/tr22/tr22-8.html Previous Version http://www.unicode.org/reports/tr22/tr22-7.html Latest Version http://www.unicode.org/reports/tr22/ DTDs http://www.unicode.org/reports/tr22/CharacterMapping.dtd http://www.unicode.org/reports/tr22/CharacterMappingAliases.dtd Revision 8 Summary This document specifies an XML format for the interchange of mapping data for character encodings, and describes some of the issues connected with the use of character conversion. It provides a complete description for such mappings in terms of a defined mapping to and from Unicode, and a description of alias tables for the interchange of mapping table names. Status This document has been reviewed by Unicode members and other interested parties, and has been approved for publication by the Unicode Consortium. This is a stable document and may be used as reference material or cited as a normative reference by other specifications. No further revisions are planned. A Unicode Technical Standard (UTS) is an independent specification. Conformance to the Unicode Standard does not imply conformance to any UTS. Please submit corrigenda and other comments with the online reporting form [Feedback]. Related information that is useful in understanding this document is found in the References. For the latest version of the Unicode Standard see [Unicode]. For a list of current Unicode Technical Reports see [Reports]. For more information about versions of the Unicode Standard, see [Versions]. Contents 1 Introduction 1.1 Illegal and Unassigned Codes 1.1.1 Best-Fit Mappings 1.1.2 Dual Substitution Handling 1.2 Completeness 1.3 Canonical Equivalence 1.4 Charset Alias Matching 2 Conformance 3 Character Mapping Table Format 3.1 Header 3.2 History 3.3 Validity Specification 3.3.1 Error Conditions 3.3.2 Simple SI/SO-Stateful Encodings 3.4 Assignments 3.4.1 Mapping Multiple Characters 3.4.2 Error Conditions 3.5 ISO 2022 4 Alias Table Format 5 Samples 5.1 Full Sample 5.2 UTF-8 Sample 5.2.1 Partial Validity Checks 5.2.2 Full Validity Checks Data Files References Modifications 1 Introduction This document has been stabilized, however the discussion of issues in this document remains relevant, although the specific XML format is not commonly used. For example, the Unicode ICU project uses conversion data files in a different format, as does the W3C Encoding specification. In addition, newer resources such as UTR #36, Unicode Security Considerations, [Unicode] Section 3.9, Unicode Encoding Forms (especially definition D93 Encoding form conversion), and [Unicode] Section 5.22, Best Practice for U+FFFD Substitution, expand on many of the issues discussed here. The ability to seamlessly handle multiple languages and writing systems character encodings is crucial in today's world, where a server may need to handle many different client languages covering many different markets. No matter how characters are represented, servers need to be able to process them appropriately. Unicode provides a common model and representation of characters for all the languages of the world. Because of this, Unicode has been adopted by all modern systems as the internal storage processing code. Rather than trying to maintain data in literally hundreds of different encodings, a program can convert the source data into Unicode on entry, process it as required, and, if needed, convert it into a target character set on request. It is vital to maintain the consistency of data across conversions between different character encodings. Because of the fluidity of data in a networked world, it is easy for it to be converted from, say, CP950 on a Windows platform, sent to a UNIX server as UTF-8, processed, and converted back to CP950 for representation on another client machine. This requires implementations to have identical mappings for a character encoding, no matter what platform they are working on. It also requires them to use the same name for the same encoding, and different names for different encodings. This is difficult to do unless there is a standard specification for the mappings so that it can be precisely determined what the encoding actually maps to. This technical report provides a standard specification for the interchange of mapping data for character encodings. By using this specification, implementations on any platform can be assured of providing precisely the same mappings as all other implementations, regardless of platform. The use of CharMapML in and of itself does not guarantee that the result of a mapping is in a Unicode Encoding Form. The DTD does not specify valid documents. It is insufficient for the specification of all of the constraints on CharMapML files. The constraints are fully specified in this Unicode Technical Standard. 1.1 Illegal and Unassigned Codes When converting data between different character encodings, the conversion software needs to distinguish the different types of errors that can occur. These fall into three main categories: sequences that are illegal, unassigned and unmappable. There are two variants when the sequence is illegal. In the first variant, the sequence is incomplete. For example, 0xA3 is incomplete in CP950. Unless followed by another byte of the right form, it is illegal. 0xC2 is incomplete in UTF-8. Unless followed by another value of the right form, it is illegal. 0x80 is incomplete in UTF-8. Unless preceded by another value of the right form, it is illegal. The second variant is where the sequence is complete, but explicitly illegal. For example, 0xC0 is illegal in UTF-8. This value can never occur in valid UTF-8 text. In the second category, the source sequence represents a valid code point, but is unassigned (also known as undefined). This sequence may be given an assignment in some future version of the character encoding. For example, 0xA3 0xBF is unassigned in CP950, as of 1999. 0x0EDE is unassigned in Unicode, V3.0 In the third category, the source sequence is assigned, but unmappable: there is no corresponding code point in the target encoding to accurately represent the source sequence. For example, the long dash is assigned in Unicode, but cannot be mapped to ISO-8859-1. In the case of illegal source sequences, a conversion routine will typically provide three options. It may stop with an error (or throw an exception). Secondly, it may skip the source sequence. While this is commonly an option, it can also hide corruption problems in the source text. Lastly, it may map to a substitution character such as the Unicode REPLACEMENT CHARACTER (U+FFFD). When a conversion routine stops with an error, the routine should communicate the cause of the error and the length and contents of the bad sequence. It should be possible to resume the conversion after the caller handles the bad sequence. There is an important difference between the case where a sequence represents a real REPLACEMENT CHARACTER in a legacy encoding, as opposed to just being unassigned, and thereby mapped to REPLACEMENT CHARACTER (using an API substitution option). An API may choose to signal an illegal sequence in a legacy character set by mapping it to a noncharacter code point (Definition D7b in the Unicode Standard), such as U+FFFF. However, this mechanism runs the risk of these values being transmitted in Unicode text (which is thus non-conformant), and should be used with caution. Unassigned sequences can be handled with any of the above options, plus some additional ones. They should always be treated as a single code point: for example, 0xA3BF is treated as a single code point when mapping into Unicode from CP950. Especially because unassigned characters may actually come from a more recent version of the character encoding, it is often important to preserve round-trip mappings if possible. This can be done by mapping to private use space. Unicode (and some other character encodings) provide a large area of Private Use characters. These can be used to provide round-trip mappings for private use characters from other character encodings, as well as provisional mappings for characters that have not yet been encoded in Unicode. A second option is to represent unassigned sequences by hex escape sequences. For example, when mapping from U+1234 to other code pages, it can be represented by "ሴ" in XML or HTML, "\u1234" in Java, C99 or C++, or "\x{1234}" in Perl. For unmappable sequences, an additional option of mapping to a fallback character sequence may be available. In this case, an unmappable sequence is given a "best fit" mapping. For example, an encoding might not have curly quotes; the generic quotes could be used as a fallback; or if EM DASH is unmappable, a sequence of two HYPHEN-MINUS characters could be used as a fallback. It is important that systems be able to distinguish between the fallback mappings and regular mappings. Systems like XML allow the use of decimal or hexadecimal escape sequences (Numeric Character References) to preserve round-trip integrity; use of fallback characters in that case corrupts the data. Because illegal sequences represent some corruption of the data stream, conversion routines may be directed to handle them differently than unassigned or unmappable sequences. For example, a routine might map an unassigned sequence to a substitution character, but throw an exception when it encounters an illegal sequence. 1.1.1 Best-Fit Mappings In cases where a specified character mapping table is not available, a best-fit mapping table can be used. This technique should be used with caution because data can be corrupted. For example, in XML there are different strategies depending on whether the process is parsing or generating. Suppose that there are two sets X and SUB_X, where X is a superset of SUB_X. (That is, every roundtrip mapping that is in SUB_X is also in X, and X may contain additional round-trip mappings.) Then: It is acceptable to parse with X when the file is tagged as SUB_X. Because X is a superset, all the characters will be read correctly. Any characters that are not in SUB_X will be encoded as NCRs (for example, ꯍ), and will work. It is acceptable to generate the file with SUB_X, and tag the file as X. Everything works as long as the characters that are not in SUB_X are converted into NCRs. It is NOT acceptable to parse with SUB_X when the file is tagged with X because characters will be corrupted. It is NOT acceptable to generate the file with X, and tag the file with SUB_X because characters will be corrupted. Therefore, looking up a best-fit character mapping needs to yield different results depending on whether a subset or a superset is required. Section 4, Alias Table Format describes data that can be used for this. 1.1.2 Dual Substitution Handling Some mapping tables for multibyte code pages define an additional, alternate code page substitution character "subchar1" which is always a single-byte code. In this case, the regular substitution character is always a double-byte code. These mapping tables then also list which unassigned code points should map to this alternate subchar1 instead of to the regular substitution character. The XML character mapping table format provides for the specification of the "subchar1" byte sequence as a sub1 attribute of the assignments element, and for the use of sub1 elements to specify which Unicode code points should map to "subchar1" instead of to the regular substitution character. Usage: In this context characters are thought of as being "wide" or "narrow." In legacy code pages, this is identified with the codes being single-byte or double-byte codes. In mappings between two legacy code pages: When a wide (double-byte) character is unassigned, it results in a double-byte substitution char…