It seems that many people aren’t as well informed about character references as they should be, so I’m going to clearly explain everything you need to know (and some things you probably don’t) about them, primarily for HTML and XML. There are two types of character references: numeric character references and character entity references.
Numeric Character References
There are two forms of numeric character references: decimal and hexadecimal.
Decimal take the form
&#nnnn;, where nnnn is
the reference to the Unicode code point for the character in decimal
Hexadecimal takes the form
&#xhhhh; where hhhh is
the code point in hexadecimal notation
character references in HTML, the
is case insensitive but in XML it must be lower case. So, for
is valid in HTML but invalid in XML.
In HTML, the reference end is either:
- a reference close (REFC) delimiter (a semicolon by default)
- a record end (RE) function code
- a character, such as a space, which is not part of a valid entity name.
That basically means that the semi-colon may be omitted from the end in certain circumstances. However, remembering exactly when it can and can’t be omitted is difficult; and to avoid the chance of error as much as possible, it’s very good practice to always include it. In XML, the semi-colon is required. These same rules apply the character entity references as well.
It is most important to note that regardless of the document’s character encoding, numeric
character references always refer to the code position in the Unicode
character repertoire .
In SGML terms, this is referred to as the Document
Character Set (DCS) and, for HTML, is defined in the SGML Declaration.
For both HTML and XML the DCS is defined as ISO-10646 (or Unicode). You
should note that there is a difference between the DCS and the
encoding, which can be anything, including
or whatever else; but the encoding does not affect numeric character
references in any way, they always refer to a character within the DCS
which is, as stated, defined as Unicode.
For example: if you want to include the right single quotation mark (’) using
a numeric character reference, you need to know the Unicode code point. In
this case, it is
8217. Thus, the hexadecimal and decimal character references
will be, respectively,
A very common mistake is to use the code point from the Windows-1252
character repertoire instead. You should be aware that this is not correct,
even though browsers have been forced to support them simply because
IE does. The
problematic code points range from 128 to 159. In Windows-1252, the right
single quotation mark falls within this range at the code position
or, more commonly known in decimal as
146. However, you cannot
include the character using this code point, as in
because doing so would actually refer to the Unicode code point, not
and characters in this range are defined as control characters.
These control characters are defined as
UNUSED in the SGML
Declaration for HTML. In confusing and obscure SGML terms, that makes them non-SGML
characters. According to the SGML handbook in section 13.1.1 that simply
meaning is assigned to that character, but explicitly states in section
a non-SGML character can be entered as a data character within
an SGML entity by using a character reference.
Strictly speaking, although these characters cannot be included as data characters within an HTML document, it is not invalid to refer to these characters with character references. The problem is a combination of the fact that the meaning of a non-SGML character is nothing short of obscure and, in Unicode, these characters are non-printable control characters.
Because of this, the validator will only issue a warning; but, although its use is still technically valid, it should be treated as an error as it almost certainly does not mean what the author intended — in fact, its meaning is undefined. To clarify the validation issues a little more, compare the results of using a non-SGML character directly within the markup and a reference to a non-SGML character. The first will fail validation with an error; while the second will pass, but with a warning, even though they’re essentially using the same character.
You should note that this is not the case for XML (including XHTML). Technically, this range (from 128 to 159) is perfectly valid according the the production for Char in XML, but they still refer to Unicode control characters, their meaning is undefined in the context of the document and thus should not be used. Although the W3C Validator will issue the same error and warning for equivalent XHTML documents, this is a symptom of its origin as an SGML validator patched to work in a sort-of XML mode. However, validating with a true XML validator (like that provided by Page Valet) will not result in any errors or warnings at all.
It is important to realise that, in XML, using or referring to a character that does not match the production for Char violates a well-formedness constraint. For example, using a control character in the range from 0 to 31 (except for tab, newline and carriage return) either directly or with a numeric character reference results in a well-formedness error.
Character Entity References
Character entity references use symbolic names instead of numbers, and take
&name;. All entity references are case
sensitive. So, for example,
to two separate characters in HTML: å and Å, respectively. The rules for
the reference end are the same as for numeric character references (discussed
the well known entity references in HTML include
" was actually removed from HTML 3.2, but
this was later realised to be a mistake and added back again in HTML 4.
Predefined Entity References in XML
XML, those are 4 of the 5 predefined entity references that can be used in
any XML document, without needing them to be being defined in a DTD. The
5th predefined entity reference in XML is
the reason I mention it separately from the others is that it is not defined
in HTML and, as a result, it is also not supported in IE for HTML. However,
it is rare that one actually needs to use it, as it is only required within
an attribute value delimited by single quotes (‘), rather than the more conventional
double quotes (“). In such cases, a numeric character reference can always be
used in its place.
External Entity References
HTML 4, XHTML 1.x and MathML define many other character entity references in their respective DTDs. These are called external entity references. In HTML, they are divided into three groups: ISO-8859-1 characters, symbols, mathematical symbols, and Greek letters and markup-significant and internationalization characters. Digital Media Minute have provided a useful character entity reference chart containing all of these. If you’re interested in the MathML entities, see chapter 6 of MathML 2.0.
Because these are defined in the DTD, technically none of them
can be used in an HTML document without an appropriate
referencing an appropriate HTML DTD; although since browsers don’t read the
DTD anyway, browsers will support them, regardless. However, in XHTML and
MathML (served with an XML MIME type), the
required for practical reasons to use any entity, other than the 5 predefined
’ are defined
in the XHTML DTD, they are not predefined in XML and so require the DTD to
be used. Without it, their use violates a well-formedness contraint, but
it should be noted that using externally defined entities is unsafe in XML because
it requires a validating XML parser to read the DTD. The Mozilla
Web Developer FAQ notes:
In older versions of Mozilla as well as in old Mozilla-based products, there is no pseudo-DTD catalog and the use of externally defined character entities (other than the five pre-defined ones) leads to an XML parsing error. There are also other XHTML user agents that do not support externally defined character entities (other than the five pre-defined ones). Since non-validating XML processors are not required to support externally defined character entities (other than the five pre-defined ones), the use of externally defined character entities (other than the five pre-defined ones) is inherently unsafe in XML documents intended for the Web. The best practice is to use straight UTF-8 instead of entities. (Numeric character references are safe, too.).
As noted, the alternative is to just use a numeric character reference instead,
but the best option is to just use a Unicode encoding, such as
and enter the real character (see my Guide
to Unicode for more information). Arguably, if you’re using a Unicode encoding,
one of the only times when it is useful to use a character reference instead
of the real character is for non-printable characters, such as non-breaking
Em-space, En-space, zero-width characters, etc. The main reason for that
is to be able to clearly identify them when you’re reading/editing the source
Numeric character references, both decimal and hexadecimal, can be safely
used in (X)HTML and XML, but you need to be careful that you’re referencing
the character’s code point from the Unicode character repertoire, not
in the range from 128 to 159).
Character entity references can be used in HTML and in XML; but for XML, other
than the 5 predefined entities, need to be defined in a DTD (such as with XHTML
and MathML). The 5 predefined entities in XML are:
Of these, you should note that
' is not defined
in HTML. The use of other entities in XML requires a validating parser, which
makes them inherently unsafe for use on the web. It is recommended that you
stick with the 5 predefined entity references and numeric character references,
or use a Unicode encoding.