{"id":57,"date":"2005-01-03T16:29:42","date_gmt":"2005-01-03T16:29:42","guid":{"rendered":"http:\/\/lachy.id.au\/log\/2005\/01\/guide-to-unicode-part-3"},"modified":"2006-04-30T23:49:59","modified_gmt":"2006-04-30T23:49:59","slug":"guide-to-unicode-part-3","status":"publish","type":"post","link":"https:\/\/lachy.id.au\/log\/2005\/01\/guide-to-unicode-part-3","title":{"rendered":"Guide to Unicode, Part 3"},"content":{"rendered":"

In the beginning we discussed character repertoires, code points and HTML<\/abbr>\r\n\tcharacter references; together with their relationship to the Unicode standard.\r\n\tWe then looked at character encodings, examining the differences between\r\n\tsingle- and multiple-octet encodings, and how to create files in UTF-8<\/code>. If you\r\n\tare unfamiliar with those concepts, I recommend that you read parts\r\n\t1<\/a>\r\n\tand 2<\/a> \r\nof this guide first and then return to this section when you are ready.<\/p>\r\n

The saga continues in this third and final thrilling chapter, where we will\r\n\tlook at some of the problems encountered with the use of Unicode and, in particular,\r\n\tUTF-8<\/code> and the BOM<\/abbr>. This will involve taking a look at the debugging of the most\r\n\tcommon problems and how the tools we\u2019ve looked at previously can help you with\r\n\tthis. Following this I will discuss the importance and practicalities of ensuring\r\n\tthat the character encoding is correctly declared. Finally I will discuss the\r\n\tpurpose of the BOM<\/abbr> and show the difference between the UTF-16<\/code> and UTF-32<\/code> variants:\r\n\tLittle Endian<\/em> and Big Endian<\/em>.<\/p>\r\n

The biggest problem with using Unicode on the web is that not all editors\r\n\tsupport Unicode, although some authors don\u2019t realise this and declare the encoding\r\n\tas UTF-8<\/code> anyway. This only becomes a problem when characters outside the US-ASCII<\/code>\tsubset are used. However, since we have already covered the creation and editing\r\n\tof Unicode files, there is no need to discuss this problem further.<\/p>\r\n

The next major issue encountered is that user agents may not always display\r\n\tthe characters correctly. Ignoring the availability of Unicode fonts for now,\r\n\tpeople will often attempt to use UTF-8<\/code>; yet find that some characters that they\r\n\thave used get turned into 2 or 3 seemingly random characters. After that they\r\n\ttend to give up and revert back to ISO-8859-1<\/code> (or Windows-1252<\/code>) and use character\r\n\treferences for the characters outside of these repertoires, or US-ASCII<\/a><\/code> substitutes<\/a>.<\/p>\r\n

For those of you that have read part 2; you should now recognise that the\r\n\tdisplayed characters are not random, but are in fact the multi-octet UTF-8<\/code> encoded\r\n\tcharacters interpreted as a single-octet encoding. This error is usually caused\r\n\tby incorrectly declaring a single-octet encoding \u2014 most often ISO-8859-1<\/code>. For\r\n\texample, the BOM<\/abbr> (if present) may be displayed as \u00ef\u00bb\u00bf<\/samp> when\r\n\ta UTF-8<\/code> file is incorrectly declared as ISO-8859-1<\/code> by the HTTP<\/abbr> response headers\r\n\t(similar to the demonstration in part 2 where the character encoding was manually\r\n\toverridden).<\/p>\r\n

For any sequence of characters appearing incorrectly like this, you may use\r\n\tIan Hickson<\/a>\u2019s UTF-8<\/a><\/code> decoder<\/a> to\r\n\tdetermine what the character is. Conversely, you may also use the UTF-8<\/code> encoder<\/a>\tto reveal both the ISO-8859-1<\/code> representation\r\n\tas well as the hexadecimal octet values for any Unicode character.<\/p>\r\n

If you suspect that you are experiencing this kind of error, then the first\r\n\tthing to check is which encoding is being used by the user agent. Most popular\r\n\tUAs will allow you to view the character encoding, such as in Mozilla\u2019s Page\r\n\tInfo dialog or Opera\u2019s Info panel. The W3C\u2019s\r\n\tMarkUp Validator<\/a> in verbose mode\r\n\twill also let you know this information. If you find that the character encoding\r\n\tis incorrect, then it is then necessary to determine whence the user agent is\r\n\tacquiring this information and to correct the error. If, however, you find that\r\n\tthe character encoding is correct, then the problem is caused elsewhere and\r\n\twill be discussed later.<\/p>\r\n

Depending on the file format, the character encoding may be declared on a\r\n\tfile-by-file basis through, for example, the use of the\r\n\t<meta><\/code> element<\/a>\r\n\tin HTML<\/abbr>,\tthe\r\n\t<?xml?><\/code> declaration<\/a>\r\n\tin XML<\/abbr> documents, the @charset<\/code>\r\n\trule<\/a> in\r\n\tCSS or indicated by the presence of the BOM<\/abbr>.\r\n\tOn a server-wide basis it depends upon the server configuration and may be\r\n\tdeclared by the charset<\/code> parameter of\r\n\tthe Content-Type<\/code> field in the HTTP<\/abbr> response\r\n\theaders. It may also be indicated by a referencing document using, for example,\r\n\tthe charset<\/code>\r\n\tattribute<\/a> in HTML<\/abbr>\r\n\tor the encoding of the referencing document itself. Finally, in the absence\r\n\tof any of these indications, some specifications define the default encoding\r\n\tthat should be used \u2013 commonly UTF-8<\/code>, UTF-16<\/code>, ISO-8859-1<\/code> or US-ASCII<\/code>.<\/p>\r\n

The order of precedence for each applicable method is defined in the relevant\r\n\tspecification for the language being used (eg. Specifying\r\n\ta Character Encoding in HTML<\/abbr> 4.01<\/a>),\r\n\toften beginning with the Content-Type<\/code> header field having the\r\n\thighest precedence and ending with the charset attribute or encoding of the\r\n\treferencing document, having the lowest. Of course, it can be fun when\r\n\tspecifications collide<\/a> on this issue, which illustrates why it is not only important to\r\n\tdeclare the character encoding appropriately, but to ensure that it is declared\r\n\tcorrectly.<\/p>\r\n

Most of those may be easily checked simply by opening the file and seeing\r\n\twhich are included. The only one you may have difficulty with is the HTTP<\/abbr> headers\r\n\tsince these are not so readily viewable without the right tool. Thankfully,\r\n\tthere are several to choose from, ranging from Mozilla<\/a>, Firefox<\/a>, or other browser\r\n\textensions like Live HTTP<\/abbr> Headers<\/a> to online tools like the W3C\u2019s HTTP<\/abbr> HEAD<\/code> Service<\/a>.<\/p>\r\n

Once you have determined the method by which the encoding is being indicated,\r\n\tit is a simple matter of correcting the error (Correcting HTTP<\/abbr> Headers will\r\n\tbe discussed later). If more than one method is used (eg. the XML<\/abbr> declaration,\r\n\tthe meta<\/code> element and\/or the HTTP<\/abbr> Content-Type<\/code> header field), you should\r\n\talso ensure that all of them indicate the same encoding. For most document formats,\r\n\tthe recommended method to use is the HTTP<\/abbr> headers. For XML<\/abbr> documents (served\r\n\tas XML<\/abbr>) however, since they are self describing file formats, it is recommended\r\n\tthat the server be configured to omit the charset<\/code> parameter from the Content-Type<\/code>\r\n\theader and that the <?xml?><\/code> declaration be used instead.<\/p>\r\n

One thing to note: even if the character encoding is being correctly declared,\r\n\tthe use of the BOM<\/abbr> may still cause problems. Although many applications do support\r\n\tthe use of the BOM<\/abbr> (namely those that support UTF-8<\/code> properly) there are still\r\n\tmany that don\u2019t (in particular, older web browsers) and require that the file\r\n\tbe saved with the BOM<\/abbr> omitted. The problem is that not every editor that supports\r\n\tUTF-8<\/code> has an option to control the output of the BOM<\/abbr>. When the file is read\r\n\tby an application that does not support UTF-8<\/code>, or the BOM<\/abbr>, these 3 bytes may\r\n\tbe interpreted as single-octet characters. For this reason, the W3C Mark Up\r\n\tValidator will issue a warning about its use, and it is recommended that the\r\n\tBOM<\/abbr> be omitted from HTML<\/abbr> documents. (Note: this does not apply to XML<\/abbr> or XHTML<\/abbr>\r\n\tdocuments served as XML<\/abbr>, since XML<\/abbr> user agents are required to support UTF-8<\/code>\r\n\tand UTF-16<\/code> fully, including the BOM<\/abbr>.)<\/p>\r\n

To modify the HTTP<\/abbr> headers,\r\n\tit is necessary to edit a server configuration file. For the Apache\r\n\tHTTPd server<\/a>, most web hosting providers will allow web content authors to use\r\n\t.htaccess files<\/a> for this, and many other purposes. Other web servers may also\r\n\toffer similar abilities, though the method may be very different. Consult your\r\n\tserver\u2019s documentation or contact your web host for information about how\r\n\tto configure the character encoding correctly. However, for those of you\r\n\tusing an Apache server, it is simply a matter of including the appropriate\r\n\tdirectives in your .htaccess files or, if you have access to it (e.g. you\r\n\tare the server admimistrator), in the httpd.conf<\/a> file. (However, most web\r\n\thosting providers do not allow general access to that file.)<\/p>\r\n

The AddDefaultCharset<\/a><\/code> and\/or AddCharset<\/a><\/code> directives may be used. For example,\r\n\tyou may set the default charset to UTF-8<\/code>, but still wish that some files use\r\n\tISO-8859-1<\/code>. These directives in either .htaccess or httpd.conf will accomplish\r\n\tthis:<\/p>\r\n\r\n

AddDefaultCharset UTF-8\r\nAddCharset ISO-8859-1 .latin1<\/code><\/pre>\r\n\r\n
The AddCharset<\/code> directive takes both a character encoding\r\n\tname and a file extension. For files encoded as ISO-8859-1<\/code>, it\r\n\tsimply requires that the file contain a\r\n\t.latin1<\/code> file extension. This feature is most useful when Content\r\n\tNegotiation<\/a> is enabled, so that the .latin1<\/code> file extension need not be included\r\n\tin the URI<\/abbr>, but still takes effect to send the correct encoding.<\/p>\r\n
Besides the fact that some user agents do not support the BOM<\/abbr>,\r\n\tyou may still encounter problems with its use in files intended to be processed\r\n\tby some server-side technology, such as PHP<\/abbr> or JSP<\/abbr>.\r\n\tFor example, Pierre\r\n\tIgot<\/a> described the problem\r\n\the encountered<\/a> with the presence of the BOM<\/abbr> in\r\n\tWordPress PHP<\/abbr> files.\r\n\tThis issue occurs because upon encountering non-whitespace character data,\r\n\tthe processor will assume that the content has begun, send out all HTTP<\/abbr> headers\r\n\tand begin to output the resultant file. If, for example, you then attempt\r\n\tto use the PHP<\/abbr>\r\n\theader()<\/code> function<\/a>, an error will be received indicating that\r\n\tit is too late to modify headers because the content has already begun.<\/p>\r\n
Other errors may not cause any server-side errors, and are thus harder to\r\n\tcatch, but they may result in invalid markup being transmitted by the server\r\n\tdue to character\r\n\tdata<\/dfn><\/a> appearing where character data is not allowed. For example,\r\n\tconsider an include file containing a set of link<\/code> elements<\/a>\tto be included within the head<\/code> element<\/a>s of several (X)HTML<\/abbr>\r\n\tfiles using some scripting language or SSI<\/abbr><\/a>:\r\n\tif the include file is encoded as UTF-8<\/code> and\r\n\tbegins with the\r\n\tBOM<\/abbr>, yet the scripting language processor\r\n\tdoes not support UTF-8<\/code> and therefore\r\n\tneither recognises the BOM<\/abbr> nor strips it\r\n\tfrom the output, it will end up being included within the head<\/code> elements.\r\n\tSince a head<\/code> element may\r\n\tnot contain character data, a markup validator will detect this and issue\r\n\tan appropriate error message.<\/p>\r\n
The exact error will differ depending on whether you are using HTML<\/abbr> or XHTML<\/abbr>,\r\n\tthough the cause will be the same. For HTML<\/abbr>, the BOM<\/abbr> will\r\n\timplicitly end the head<\/code> element,\r\n\tand begin the body<\/code>\telement<\/a>. Thus, anything following will be treated\r\n\tas being within the body<\/code> element, not the head<\/code>, which may cause several errors\r\n\tdepending on the content. For XHTML<\/abbr>, however, the BOM<\/abbr> will simply be treated\r\n\tas character data appearing within the head element, where no character data\r\n\tis allowed.<\/p>\r\n
The final problem that I have encountered is slightly more complicated and\r\n\tproved quite difficult to solve. While using JSP<\/abbr> to\r\n\tdevelop a web site, I had encoded all files as UTF-8<\/code>, omitting\r\n\tthe BOM<\/abbr>. The issue was that one include\r\n\tfile included the copyright symbol (U+00A9<\/a>),\r\n\twhich is encoded in UTF-8<\/code> as the\r\n\toctets C2 A9<\/code>. At first, it appeared as though the UA was interpreting the\r\n\tcharacter as a single octet encoding, thus displaying the characters \u00c2\u00a9<\/samp>.\r\n\tHowever, the HTML<\/abbr> document was\r\n\tbeing correctly interpreted as UTF-8<\/code>, since the\r\n\tcharacter encoding was being declared by the HTTP<\/abbr> Content-Type<\/code> header\r\n\tfield.<\/p>\r\n
After much investigation, I found that if I encoded\r\n\tthe include file as ISO-8859-1<\/code>, but the main file as UTF-8<\/code>, the desired UTF-8<\/code>\r\n\toutput was received. It turned out that the JSP<\/abbr> processor, Apache Tomcat with\r\n\tJBoss, thought that the include file was to be interpreted as ISO-8859-1<\/code> (the\r\n\tdefault for JSP<\/abbr>); however the output was required to be UTF-8<\/code>. Because of this,\r\n\tthe JSP<\/abbr> processor was attempting to convert the character encoding of the include\r\n\tfile into UTF-8<\/code> on the fly.<\/p>\r\n
Thus, when it encountered the octets C2 A9<\/code>, it interpreted them as ISO-8859-1<\/code>\r\n\tcharacters, which map to the Unicode characters: U+00C2 and U+00A9. These characters,\r\n\twhen encoded as UTF-8<\/code>, form the octets C3 82<\/code> and C2\r\n\tA9<\/code>, respectively, which\r\n\tis the output I was receiving in the HTML<\/abbr> document. I ended up solving this\r\n\tproblem by correctly informing the JSP<\/abbr> processor that the include files were\r\n\talso encoded as UTF-8<\/code>, and not the default ISO-8859-1<\/code>.<\/p>\r\n
Up until now, we have looked at the BOM<\/abbr>, discussed how it is encoded in UTF-8<\/code>\r\n\tand some of the problems it may cause; however we have not looked at its purpose.\r\n\tAs mentioned, the BOM<\/abbr> is optional in UTF-8<\/code>; but it is required in UTF-16<\/code> and\r\n\tUTF-32<\/code> to indicate\r\n\tthe order of octets<\/a> for each character as either Little\r\n\tEndian<\/dfn><\/a>,\r\n\twhere the least significant byte appears first, and Big\r\n\tEndian<\/dfn><\/a> where the most\r\n\tsignificant byte appears first. For this guide, we will only look at UTF-16<\/code>,\r\n\tbut a similar technique still applies to UTF-32<\/code> documents.<\/p>\r\n
 In UTF-16LE<\/code> the BOM<\/abbr> (U+FEFF<\/a>) will appear in the file as the octet sequence\r\n\tFF FE<\/code> (least significant byte first), but in UTF-16BE<\/code> it appears as FE FF<\/code> (most\r\n\tsignificant byte first). For UTF-16<\/code>, because the reversed sequence U+FFFE is defined never\r\n\tto be a character in Unicode, that sequence of octets (FF FE<\/code>) can be safely\r\n\tused to detect the encoding as UTF-16LE<\/code>, and not a UTF-16BE<\/code> file\r\n\tstarting with the character U+FFFE, or vice versa.<\/p>\r\n
Now that you have a rather more in-depth understanding of Unicode (including:\r\n\tthe character repertoire, code points and the various encoding forms; the\r\n\tability to create and edit Unicode encoded files; and an understanding\r\n\tof several of the problems that may be encountered) it is time to go forth\r\n\tand prosper \u2014 to make use of Unicode to its full potential, to simplify the\r\n\tuse of non-US-ASCII<\/code> chracters and to help promote the i18n<\/abbr>\r\n\tof the web<\/a>.  However,\r\n\tas with everything else<\/em>, there\u2019s always more to be learned. Although\r\n\tit may seem that I have covered much, I\u2019m sure you\r\n\twill find that I have only just scratched the surface. So, to help you out,\r\n\tI\u2019ve compiled a short but comprehensive list of additional resources that\r\n\twill provide further information:<\/p>\r\n\t