{"id":90,"date":"2005-10-10T14:36:58","date_gmt":"2005-10-10T14:36:58","guid":{"rendered":"http:\/\/lachy.id.au\/log\/2005\/10\/char-refs"},"modified":"2006-10-19T06:37:11","modified_gmt":"2006-10-19T06:37:11","slug":"char-refs","status":"publish","type":"post","link":"https:\/\/lachy.id.au\/log\/2005\/10\/char-refs","title":{"rendered":"Character References Explained"},"content":{"rendered":"<p>It seems that many people aren&#8217;t as well informed about character references<br \/>\n\tas they should be, so I&#8217;m going to clearly explain everything you need to<br \/>\n\tknow (and some things you probably don&#8217;t) about them, primarily for HTML<br \/>\n\tand XML. There are two types of character references: <a href=\"#charref-numeric\">numeric<br \/>\n\tcharacter references<\/a> and <a href=\"#charref-entity\">character entity references<\/a>.<\/p>\n<h3 id=\"charref-numeric\">Numeric Character References<\/h3>\n<p>There are two forms of numeric character references: decimal and hexadecimal.<\/p>\n<h4 id=\"charref-numeric-syntax\">Syntax<\/h4>\n<p>Decimal take the form <code>&amp;#<var>nnnn<\/var>;<\/code>, where <var>nnnn<\/var> is<br \/>\n\tthe reference to the Unicode code point for the character in decimal <code>[0-9]<\/code>.<br \/>\n\tHexadecimal takes the form <code>&amp;#x<var>hhhh<\/var>;<\/code> where <var>hhhh<\/var> is<br \/>\nthe code point in hexadecimal notation <code>[0-9a-fA-F]<\/code>.<\/p>\n<p> For hexadecimal<br \/>\n\tcharacter references in HTML, the <code>x<\/code><br \/>\n\tis case insensitive but in XML it <strong>must<\/strong> be lower case. So, for<br \/>\n\texample, <code>&amp;#XA0;<\/code> is valid in HTML but invalid in XML.<\/p>\n<p>In HTML, the <dfn><a href=\"http:\/\/www.is-thought.co.uk\/book\/sgml-6.htm#refc\">reference end<\/a><\/dfn> is either:<\/p>\n<ul>\n<li>a reference close (REFC) delimiter (a semicolon by default) <\/li>\n<li>a record end (RE) function code <\/li>\n<li>a character, such as a space, which is not part of a valid entity name.<\/li>\n<\/ul>\n<p>That basically means that the semi-colon may be omitted from the end in certain<br \/>\n\tcircumstances. However, remembering exactly when it can and can&#8217;t be omitted<br \/>\n\tis difficult; and to avoid the chance of error as much as possible, it&#8217;s<br \/>\n\tvery good practice to always include it. In XML, the semi-colon is required.<br \/>\n\tThese same rules apply the character entity references as well.<\/p>\n<h4 id=\"charref-numeric-repertoire\">Character Repertoire <\/h4>\n<p>It is most important to note that regardless of the document&#8217;s character encoding, <strong>numeric<br \/>\n\t\tcharacter references <em>always<\/em> refer to the code position in the Unicode<br \/>\n\t\tcharacter repertoire <\/strong>.<br \/>\n\t\tIn SGML terms, this is referred to as the <a href=\"http:\/\/www.w3.org\/TR\/html401\/charset.html#h-5.1\">Document<br \/>\n\t\tCharacter Set<\/a> (DCS) and, for HTML, is defined in the SGML Declaration.<br \/>\n\t\tFor both HTML and XML the DCS is defined as ISO-10646 (or Unicode). You<br \/>\n\t\tshould note that there is a difference between the DCS and the<br \/>\n\t\tfile&#8217;s <a href=\"http:\/\/www.w3.org\/TR\/html401\/charset.html#h-5.2\">character<br \/>\n\t\tencoding<\/a>, which can be anything, including <code>ISO-8859-1<\/code>, <code>UTF-8<\/code>, <code>UTF-16<\/code>, <code>Shift_JIS<\/code>,<br \/>\n\t\tor whatever else; but the encoding <em>does not<\/em> affect numeric character<br \/>\n\t\treferences in any way, they always refer to a character within the DCS<br \/>\n\t\twhich is, as stated, defined as Unicode.<\/p>\n<p>For example: if you want to include the right single quotation mark (\u2019) using<br \/>\n\ta numeric character reference, you need to know the Unicode code point. In<br \/>\n\tthis case, it is <code>U+2019<\/code> or,<br \/>\n\tin decimal, <code>8217<\/code>. Thus, the hexadecimal and decimal character references<br \/>\n\twill be, respectively, <code>&amp;#x2019;<\/code> and <code>&amp;#8217;<\/code>.<\/p>\n<h4 id=\"charref-numeric-mistake\">Common Mistakes<\/h4>\n<p>A very common mistake is to use the code point from the <a href=\"http:\/\/www.unicode.org\/Public\/MAPPINGS\/VENDORS\/MICSFT\/WINDOWS\/CP1252.TXT\">Windows-1252<br \/>\n\t\tcharacter repertoire<\/a> instead. You should be aware that this is not correct,<br \/>\n\t\teven though browsers have been forced to support them simply because<br \/>\n\t\tIE does. <a href=\"http:\/\/www.cs.tut.fi\/%7Ejkorpela\/www\/windows-chars.html#list\">The<br \/>\n\t\tproblematic code points range from 128 to 159<\/a>. In Windows-1252, the right<br \/>\n\t\tsingle quotation mark falls within this range at the code position <code>0x92<\/code>,<br \/>\n\t\tor, more commonly known in decimal as <code>146<\/code>. However, you cannot<br \/>\n\t\tinclude the character using this code point, as in <code>&amp;#x92;<\/code> or <code>&amp;#146;<\/code>,<br \/>\n\t\tbecause doing so would actually refer to the Unicode code point, not <code>Windows-1252<\/code>,<br \/>\n\t\tand characters in this range are defined as control characters.<\/p>\n<p>These control characters are defined as <code>UNUSED<\/code> in the <a href=\"http:\/\/www.w3.org\/TR\/html401\/sgml\/sgmldecl.html\">SGML<br \/>\n\t\tDeclaration for HTML<\/a>. In confusing and obscure SGML terms, that makes them <dfn>non-SGML<br \/>\n\t\tcharacters<\/dfn>. According to the SGML handbook in section 13.1.1 that simply<br \/>\n\t\tmeans that <q>no<br \/>\n\t\tmeaning is assigned to that character<\/q>, but explicitly states in section<br \/>\n\t\t9.2 that <q>a non-SGML character can be entered as a data character within<br \/>\n\t\tan SGML entity by using a character reference<\/q>.<\/p>\n<h4 id=\"charref-numeric-valid\">Validation Issues<\/h4>\n<p>Strictly speaking, although these characters cannot be included<br \/>\n\tas data characters within an HTML document, it is <strong>not invalid<\/strong> to<br \/>\n\trefer to these characters with character references. The problem is a<br \/>\n\tcombination of the fact that the meaning of a non-SGML character is nothing<br \/>\n\tshort of obscure and, in Unicode, these characters are non-printable control<br \/>\n\tcharacters.<\/p>\n<p>Because of this, the validator will only issue a warning; but, although<br \/>\n\tits use is still <em>technically valid<\/em>, it should be <strong>treated as<br \/>\n\tan error<\/strong> as it almost certainly <strong>does not mean what the<br \/>\n\tauthor intended<\/strong> \u2014 in fact, <strong><em>its meaning is undefined<\/em><\/strong>.<br \/>\n\tTo clarify the validation issues a little more, compare the results<br \/>\n\tof using a <a href=\"http:\/\/validator.w3.org\/check?uri=data%3Atext%2Fhtml%3Bcharset%3Dutf-8%2C%253C%21DOCTYPE%2520HTML%2520PUBLIC%2520%2522-%252F%252FW3C%252F%252FDTD%2520HTML%25204.01%252F%252FEN%2522%253E%250D%250A%253Ctitle%253Enon%2520SGML%2520character%2520number%2520146%253C%252Ftitle%253E%250D%250A%253Cp%253E%25C2%2592&amp;charset=%28detect+automatically%29&amp;doctype=Inline&amp;ss=1&amp;verbose=1\">non-SGML<br \/>\n\tcharacter directly within the markup<\/a>\tand<br \/>\n\t<a href=\"http:\/\/validator.w3.org\/check?uri=data%3Atext%2Fhtml%3Bcharset%3Dutf-8%2C%253C%21DOCTYPE%2520HTML%2520PUBLIC%2520%2522-%252F%252FW3C%252F%252FDTD%2520HTML%25204.01%252F%252FEN%2522%253E%250D%250A%253Ctitle%253Ereference%2520to%2520non-SGML%2520character%253C%252Ftitle%253E%250D%250A%253Cp%253E%2526%2523146%253B&amp;charset=%28detect+automatically%29&amp;doctype=Inline&amp;ss=1&amp;verbose=1\">a<br \/>\n\treference to a non-SGML character<\/a>. The first will fail validation with<br \/>\n\tan <strong>error;<\/strong> while the second will pass, but with a <strong>warning<\/strong>,<br \/>\n\teven though they&#8217;re essentially using the same character.<\/p>\n<p>You should  note that this is not the case for XML (including XHTML). Technically,<br \/>\n\tthis range (from 128 to 159) is perfectly valid according the the <a href=\"http:\/\/www.w3.org\/TR\/REC-xml\/#NT-Char\">production<br \/>\n\t\tfor Char in XML<\/a>, but they still refer to Unicode control characters, their<br \/>\n\t\tmeaning is undefined in the context of the document and thus <strong>should<br \/>\n\t\tnot<\/strong> be used. Although the W3C Validator will issue the same error<br \/>\n\t\tand warning for equivalent XHTML documents, this is a symptom of its<br \/>\n\t\torigin as an SGML validator patched to work in a sort-of XML mode. However,<br \/>\n\t\tvalidating with a true XML validator (like that provided by <a href=\"http:\/\/valet.webthing.com\/page\/\">Page<br \/>\n\t\tValet<\/a>) will not result in any errors or warnings at all.<\/p>\n<p>It is important to realise that, in XML, using or referring to a character<br \/>\n\tthat does not match the production for Char violates a well-formedness constraint.<br \/>\n\tFor example, using a control character in the range from 0 to 31 (except<br \/>\n\tfor tab, newline and carriage return) either <a href=\"data:application\/xml,%3Cp%3E%01%3C%2Fp%3E\" title=\"Sample XML document containing a control character\">directly<\/a> or<br \/>\n\t<a href=\"data:application\/xml,%3Cp%3E%26%231%3B%3C%2Fp%3E\" title=\"Sample XML document containing a numeric character reference to a control character\">with<br \/>\n\ta numeric character reference<\/a><br \/>\n\tresults in a well-formedness error.<\/p>\n<h3 id=\"charref-entity\">Character Entity References<\/h3>\n<p>Character entity references use symbolic names instead of numbers, and take<br \/>\n\tthe form <code>&amp;<var>name<\/var>;<\/code>. All entity references are case<br \/>\n\tsensitive. So, for example, <code>&amp;aring;<\/code> and <code>&amp;Aring;<\/code> refer<br \/>\n\tto two separate characters in HTML: \u00e5 and \u00c5, respectively. The rules for<br \/>\n\tthe <a href=\"http:\/\/www.is-thought.co.uk\/book\/sgml-6.htm#refc\">reference end<\/a> are the same as for numeric character references (discussed<br \/>\n\tabove). <\/p>\n<p> Some of<br \/>\n\tthe well known entity references in HTML include <code>&amp;amp;<\/code>, <code>&amp;lt;<\/code>, <code>&amp;gt;<\/code> and <code>&amp;quot;<\/code>.<br \/>\n\tInterestingly, <code>&amp;quot;<\/code> was actually removed from HTML 3.2, but<br \/>\n\tthis was later realised to be a mistake and added back again in HTML 4.<\/p>\n<h4 id=\"charref-entity-predef\">Predefined Entity References in XML <\/h4>\n<p>In<br \/>\n\tXML, those are 4 of the 5 predefined entity references that can be used in<br \/>\n\tany XML document, without needing them to be being defined in a DTD. The<br \/>\n\t5th predefined entity reference in XML is <code>&amp;apos;<\/code>, but<br \/>\n\tthe reason I mention it separately from the others is that it is not defined<br \/>\n\tin HTML and, as a result, it is also not supported in IE for HTML. However,<br \/>\n\tit is rare that one actually needs to use it, as it is only required within<br \/>\n\tan attribute value delimited by single quotes (&#8216;), rather than the more conventional<br \/>\n\tdouble quotes (&#8220;). In such cases, a numeric character reference can always be<br \/>\n\tused in its place.<\/p>\n<h4 id=\"charref-entity-ext\">External Entity References<\/h4>\n<p>HTML 4, XHTML 1.x and MathML define many other character entity references<br \/>\n\tin their respective DTDs. These are called <dfn>external entity references<\/dfn>. In<br \/>\n\tHTML, they are divided into three groups: <a href=\"http:\/\/www.w3.org\/TR\/html401\/sgml\/entities.html#h-24.2\">ISO-8859-1<br \/>\n\tcharacters<\/a>, <a href=\"http:\/\/www.w3.org\/TR\/html401\/sgml\/entities.html#h-24.3\">symbols,<br \/>\n\tmathematical symbols, and Greek letters<\/a> and <a href=\"http:\/\/www.w3.org\/TR\/html401\/sgml\/entities.html#h-24.4\">markup-significant<br \/>\n\tand internationalization characters<\/a>. Digital Media Minute have provided<br \/>\n\ta useful <a href=\"http:\/\/www.digitalmediaminute.com\/reference\/entity\/index.php\">character<br \/>\n\tentity reference chart<\/a> containing all of these. If you&#8217;re interested in<br \/>\n\tthe MathML entities, see <a href=\"http:\/\/www.w3.org\/TR\/2003\/REC-MathML2-20031021\/chapter6.html\" title=\"Characters, Entities and Fonts\">chapter<br \/>\n\t6 of MathML 2.0<\/a>.<\/p>\n<p>Because these are defined in the DTD, technically none of them<br \/>\n\tcan be used in an HTML document without an appropriate <code>DOCTYPE<\/code> declaration<br \/>\n\treferencing an appropriate HTML DTD; although since browsers don&#8217;t read the<br \/>\n\tDTD anyway, browsers will support them, regardless. However, in XHTML and<br \/>\n\tMathML (served with an XML MIME type), the <code>DOCTYPE<\/code> is<br \/>\n\trequired for practical reasons to use any entity, other than the 5 predefined<br \/>\n\tones.<\/p>\n<p>For example, <code>&amp;nbsp;<\/code> and <code>&amp;rsquo;<\/code> are defined<br \/>\n\tin the XHTML DTD, they are not predefined in XML and so require the DTD to<br \/>\n\tbe used. Without it, their use violates a well-formedness contraint, but<br \/>\n\tit should be noted that using externally defined entities is unsafe in XML because<br \/>\n\tit requires a validating XML parser to read the DTD. The <a href=\"http:\/\/www.mozilla.org\/docs\/web-developer\/faq.html#xhtmldiff\">Mozilla<br \/>\n\tWeb Developer FAQ notes<\/a>:<\/p>\n<blockquote cite=\"http:\/\/www.mozilla.org\/docs\/web-developer\/faq.html#xhtmldiff\">\n<p>In older versions of Mozilla as well as in old Mozilla-based products, there<br \/>\n\t\tis no pseudo-DTD catalog and the use of externally defined character entities<br \/>\n\t\t(other than the five pre-defined ones) leads to an XML parsing error. There<br \/>\n\t\tare also other XHTML user agents that do not support externally defined character<br \/>\n\t\tentities (other than the five pre-defined ones). Since non-validating XML processors<br \/>\n\t\tare not required to support externally defined character entities (other than<br \/>\n\t\tthe five pre-defined ones), the use of externally defined character entities<br \/>\n\t\t(other than the five pre-defined ones) is inherently unsafe in XML documents<br \/>\n\t\tintended for the Web. The best practice is to use straight UTF-8 instead of<br \/>\n\t\tentities. (Numeric character references are safe, too.).<\/p>\n<\/blockquote>\n<p>As noted, the alternative is to just use a numeric character reference instead,<br \/>\n\tbut the best option is to just use a Unicode encoding, such as <code>UTF-8<\/code> or <code>UTF-16<\/code>,<br \/>\n\tand enter the real character (see my <a href=\"\/log\/2004\/12\/guide-to-unicode-part-1\">Guide<br \/>\n\tto Unicode<\/a> for more information). Arguably, if you&#8217;re using a Unicode encoding,<br \/>\n\tone of the only times when it is useful to use a character reference instead<br \/>\n\tof the real character is for non-printable characters, such as non-breaking<br \/>\n\tspace (<code>&amp;nbsp;<\/code> or, preferably, <code>&amp;#xA0;<\/code>),<br \/>\n\tEm-space, En-space, zero-width characters, etc. The main reason for that<br \/>\n\tis to be able to clearly identify them when you&#8217;re reading\/editing the source<br \/>\n\tcode.<\/p>\n<h3 id=\"charref-summary\">Summary<\/h3>\n<p>Numeric character references, both decimal and hexadecimal, can be safely<br \/>\n\tused in (X)HTML and XML, but you need to be careful that you&#8217;re referencing<br \/>\n\tthe character&#8217;s code point from the Unicode character repertoire, not <code>Windows-1252<\/code> (especially<br \/>\n\tin the range from 128 to 159).<\/p>\n<p>Character entity references can be used in HTML and in XML; but for XML, other<br \/>\n\tthan the 5 predefined entities, need to be defined in a DTD (such as with XHTML<br \/>\n\tand MathML). The 5 predefined entities in XML are: <code>&amp;amp;<\/code>, <code>&amp;lt;<\/code>, <code>&amp;gt;<\/code>, <code>&amp;quot;<\/code> and <code>&amp;apos;<\/code>.<br \/>\n\tOf these, you should note that <code>&amp;apos;<\/code> is <em>not<\/em> defined<br \/>\n\tin HTML. The use of other entities in XML requires a validating parser, which<br \/>\n\tmakes them inherently unsafe for use on the web. It is recommended that you<br \/>\n\tstick with the 5 predefined entity references and numeric character references,<br \/>\n\tor use a Unicode encoding.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>It seems that many people aren&#8217;t as well informed about character references as they should be, so I&#8217;m going to clearly explain everything you need to know (and some things you probably don&#8217;t) about them, primarily for HTML and XML.<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":[],"categories":[14,2],"tags":[],"_links":{"self":[{"href":"https:\/\/lachy.id.au\/log\/wp-json\/wp\/v2\/posts\/90"}],"collection":[{"href":"https:\/\/lachy.id.au\/log\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/lachy.id.au\/log\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/lachy.id.au\/log\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/lachy.id.au\/log\/wp-json\/wp\/v2\/comments?post=90"}],"version-history":[{"count":0,"href":"https:\/\/lachy.id.au\/log\/wp-json\/wp\/v2\/posts\/90\/revisions"}],"wp:attachment":[{"href":"https:\/\/lachy.id.au\/log\/wp-json\/wp\/v2\/media?parent=90"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/lachy.id.au\/log\/wp-json\/wp\/v2\/categories?post=90"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/lachy.id.au\/log\/wp-json\/wp\/v2\/tags?post=90"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}