Guide to Unicode, Part 3

Characters, Standards

In the beginning we discussed character repertoires, code points and HTML character references; together with their relationship to the Unicode standard. We then looked at character encodings, examining the differences between single- and multiple-octet encodings, and how to create files in UTF-8. If you are unfamiliar with those concepts, I recommend that you read parts 1 and 2 of this guide first and then return to this section when you are ready.

The saga continues in this third and final thrilling chapter, where we will look at some of the problems encountered with the use of Unicode and, in particular, UTF-8 and the BOM. This will involve taking a look at the debugging of the most common problems and how the tools we’ve looked at previously can help you with this. Following this I will discuss the importance and practicalities of ensuring that the character encoding is correctly declared. Finally I will discuss the purpose of the BOM and show the difference between the UTF-16 and UTF-32 variants: Little Endian and Big Endian.

The biggest problem with using Unicode on the web is that not all editors support Unicode, although some authors don’t realise this and declare the encoding as UTF-8 anyway. This only becomes a problem when characters outside the US-ASCII subset are used. However, since we have already covered the creation and editing of Unicode files, there is no need to discuss this problem further.

The next major issue encountered is that user agents may not always display the characters correctly. Ignoring the availability of Unicode fonts for now, people will often attempt to use UTF-8; yet find that some characters that they have used get turned into 2 or 3 seemingly random characters. After that they tend to give up and revert back to ISO-8859-1 (or Windows-1252) and use character references for the characters outside of these repertoires, or US-ASCII substitutes.

For those of you that have read part 2; you should now recognise that the displayed characters are not random, but are in fact the multi-octet UTF-8 encoded characters interpreted as a single-octet encoding. This error is usually caused by incorrectly declaring a single-octet encoding — most often ISO-8859-1. For example, the BOM (if present) may be displayed as  when a UTF-8 file is incorrectly declared as ISO-8859-1 by the HTTP response headers (similar to the demonstration in part 2 where the character encoding was manually overridden).

For any sequence of characters appearing incorrectly like this, you may use Ian Hickson’s UTF-8 decoder to determine what the character is. Conversely, you may also use the UTF-8 encoder to reveal both the ISO-8859-1 representation as well as the hexadecimal octet values for any Unicode character.

If you suspect that you are experiencing this kind of error, then the first thing to check is which encoding is being used by the user agent. Most popular UAs will allow you to view the character encoding, such as in Mozilla’s Page Info dialog or Opera’s Info panel. The W3C’s MarkUp Validator in verbose mode will also let you know this information. If you find that the character encoding is incorrect, then it is then necessary to determine whence the user agent is acquiring this information and to correct the error. If, however, you find that the character encoding is correct, then the problem is caused elsewhere and will be discussed later.

Depending on the file format, the character encoding may be declared on a file-by-file basis through, for example, the use of the <meta> element in HTML, the <?xml?> declaration in XML documents, the @charset rule in CSS or indicated by the presence of the BOM. On a server-wide basis it depends upon the server configuration and may be declared by the charset parameter of the Content-Type field in the HTTP response headers. It may also be indicated by a referencing document using, for example, the charset attribute in HTML or the encoding of the referencing document itself. Finally, in the absence of any of these indications, some specifications define the default encoding that should be used – commonly UTF-8, UTF-16, ISO-8859-1 or US-ASCII.

The order of precedence for each applicable method is defined in the relevant specification for the language being used (eg. Specifying a Character Encoding in HTML 4.01), often beginning with the Content-Type header field having the highest precedence and ending with the charset attribute or encoding of the referencing document, having the lowest. Of course, it can be fun when specifications collide on this issue, which illustrates why it is not only important to declare the character encoding appropriately, but to ensure that it is declared correctly.

Most of those may be easily checked simply by opening the file and seeing which are included. The only one you may have difficulty with is the HTTP headers since these are not so readily viewable without the right tool. Thankfully, there are several to choose from, ranging from Mozilla, Firefox, or other browser extensions like Live HTTP Headers to online tools like the W3C’s HTTP HEAD Service.

Once you have determined the method by which the encoding is being indicated, it is a simple matter of correcting the error (Correcting HTTP Headers will be discussed later). If more than one method is used (eg. the XML declaration, the meta element and/or the HTTP Content-Type header field), you should also ensure that all of them indicate the same encoding. For most document formats, the recommended method to use is the HTTP headers. For XML documents (served as XML) however, since they are self describing file formats, it is recommended that the server be configured to omit the charset parameter from the Content-Type header and that the <?xml?> declaration be used instead.

One thing to note: even if the character encoding is being correctly declared, the use of the BOM may still cause problems. Although many applications do support the use of the BOM (namely those that support UTF-8 properly) there are still many that don’t (in particular, older web browsers) and require that the file be saved with the BOM omitted. The problem is that not every editor that supports UTF-8 has an option to control the output of the BOM. When the file is read by an application that does not support UTF-8, or the BOM, these 3 bytes may be interpreted as single-octet characters. For this reason, the W3C Mark Up Validator will issue a warning about its use, and it is recommended that the BOM be omitted from HTML documents. (Note: this does not apply to XML or XHTML documents served as XML, since XML user agents are required to support UTF-8 and UTF-16 fully, including the BOM.)

To modify the HTTP headers, it is necessary to edit a server configuration file. For the Apache HTTPd server, most web hosting providers will allow web content authors to use .htaccess files for this, and many other purposes. Other web servers may also offer similar abilities, though the method may be very different. Consult your server’s documentation or contact your web host for information about how to configure the character encoding correctly. However, for those of you using an Apache server, it is simply a matter of including the appropriate directives in your .htaccess files or, if you have access to it (e.g. you are the server admimistrator), in the httpd.conf file. (However, most web hosting providers do not allow general access to that file.)

The AddDefaultCharset and/or AddCharset directives may be used. For example, you may set the default charset to UTF-8, but still wish that some files use ISO-8859-1. These directives in either .htaccess or httpd.conf will accomplish this:

AddDefaultCharset UTF-8
AddCharset ISO-8859-1 .latin1

The AddCharset directive takes both a character encoding name and a file extension. For files encoded as ISO-8859-1, it simply requires that the file contain a .latin1 file extension. This feature is most useful when Content Negotiation is enabled, so that the .latin1 file extension need not be included in the URI, but still takes effect to send the correct encoding.

Besides the fact that some user agents do not support the BOM, you may still encounter problems with its use in files intended to be processed by some server-side technology, such as PHP or JSP. For example, Pierre Igot described the problem he encountered with the presence of the BOM in WordPress PHP files. This issue occurs because upon encountering non-whitespace character data, the processor will assume that the content has begun, send out all HTTP headers and begin to output the resultant file. If, for example, you then attempt to use the PHP header() function, an error will be received indicating that it is too late to modify headers because the content has already begun.

Other errors may not cause any server-side errors, and are thus harder to catch, but they may result in invalid markup being transmitted by the server due to character data appearing where character data is not allowed. For example, consider an include file containing a set of link elements to be included within the head elements of several (X)HTML files using some scripting language or SSI: if the include file is encoded as UTF-8 and begins with the BOM, yet the scripting language processor does not support UTF-8 and therefore neither recognises the BOM nor strips it from the output, it will end up being included within the head elements. Since a head element may not contain character data, a markup validator will detect this and issue an appropriate error message.

The exact error will differ depending on whether you are using HTML or XHTML, though the cause will be the same. For HTML, the BOM will implicitly end the head element, and begin the body element. Thus, anything following will be treated as being within the body element, not the head, which may cause several errors depending on the content. For XHTML, however, the BOM will simply be treated as character data appearing within the head element, where no character data is allowed.

The final problem that I have encountered is slightly more complicated and proved quite difficult to solve. While using JSP to develop a web site, I had encoded all files as UTF-8, omitting the BOM. The issue was that one include file included the copyright symbol (U+00A9), which is encoded in UTF-8 as the octets C2 A9. At first, it appeared as though the UA was interpreting the character as a single octet encoding, thus displaying the characters ©. However, the HTML document was being correctly interpreted as UTF-8, since the character encoding was being declared by the HTTP Content-Type header field.

After much investigation, I found that if I encoded the include file as ISO-8859-1, but the main file as UTF-8, the desired UTF-8 output was received. It turned out that the JSP processor, Apache Tomcat with JBoss, thought that the include file was to be interpreted as ISO-8859-1 (the default for JSP); however the output was required to be UTF-8. Because of this, the JSP processor was attempting to convert the character encoding of the include file into UTF-8 on the fly.

Thus, when it encountered the octets C2 A9, it interpreted them as ISO-8859-1 characters, which map to the Unicode characters: U+00C2 and U+00A9. These characters, when encoded as UTF-8, form the octets C3 82 and C2 A9, respectively, which is the output I was receiving in the HTML document. I ended up solving this problem by correctly informing the JSP processor that the include files were also encoded as UTF-8, and not the default ISO-8859-1.

Up until now, we have looked at the BOM, discussed how it is encoded in UTF-8 and some of the problems it may cause; however we have not looked at its purpose. As mentioned, the BOM is optional in UTF-8; but it is required in UTF-16 and UTF-32 to indicate the order of octets for each character as either Little Endian, where the least significant byte appears first, and Big Endian where the most significant byte appears first. For this guide, we will only look at UTF-16, but a similar technique still applies to UTF-32 documents.

In UTF-16LE the BOM (U+FEFF) will appear in the file as the octet sequence FF FE (least significant byte first), but in UTF-16BE it appears as FE FF (most significant byte first). For UTF-16, because the reversed sequence U+FFFE is defined never to be a character in Unicode, that sequence of octets (FF FE) can be safely used to detect the encoding as UTF-16LE, and not a UTF-16BE file starting with the character U+FFFE, or vice versa.

Now that you have a rather more in-depth understanding of Unicode (including: the character repertoire, code points and the various encoding forms; the ability to create and edit Unicode encoded files; and an understanding of several of the problems that may be encountered) it is time to go forth and prosper — to make use of Unicode to its full potential, to simplify the use of non-US-ASCII chracters and to help promote the i18n of the web. However, as with everything else, there’s always more to be learned. Although it may seem that I have covered much, I’m sure you will find that I have only just scratched the surface. So, to help you out, I’ve compiled a short but comprehensive list of additional resources that will provide further information: