5 The universal charset

Standard ISO 10646 defines a universal character set, intended to encompass in the long run all languages written on this planet. It is based on wide characters, and offer possibilities for two billion characters (2^31).

This charset was to become available in recode under the name UCS, with many external surfaces for it. But in the current version, only surfaces of UCS are offered, each presented as a genuine charset rather than a surface. Such surfaces are only meaningful for the UCS charset, so it is not that useful to draw a line between the surfaces and the only charset to which they may apply.

UCS stands for Universal Character Set. UCS-2 and UCS-4 are fixed length encodings, using two or four bytes per character respectively. UTF stands for UCS Transformation Format, and are variable length encodings dedicated to UCS. UTF-1 was based on ISO 2022, it did not succeed⁹. UTF-2 replaced it, it has been called UTF-FSS (File System Safe) in Unicode or Plan9 context, but is better known today as UTF-8. To complete the picture, there is UTF-16 based on 16 bits bytes, and UTF-7 which is meant for transmissions limited to 7-bit bytes. Most often, one might see UTF-8 used for external storage, and UCS-2 used for internal storage.

When recode is producing any representation of UCS, it uses the replacement character U+FFFD for any valid character which is not representable in the goal charset¹⁰. This happens, for example, when UCS-2 is not capable to echo a wide UCS-4 character, or for a similar reason, an UTF-8 sequence using more than three bytes. The replacement character is meant to represent an existing character. So, it is never produced to represent an invalid sequence or ill-formed character in the input text. In such cases, recode just gets rid of the noise, while taking note of the error in its usual ways.

Even if UTF-8 is an encoding, really, it is the encoding of a single character set, and nothing else. It is useful to distinguish between an encoding (a surface within recode) and a charset, but only when the surface may be applied to several charsets. Specifying a charset is a bit simpler than specifying a surface in a recode request. There would not be a practical advantage at imposing a more complex syntax to recode users, when it is simple to assimilate UTF-8 to a charset. Similar considerations apply for UCS-2, UCS-4, UTF-16 and UTF-7. These are all considered to be charsets.

5.1 Universal Character Set, 2 bytes

One surface of UCS is usable for the subset defined by its first sixty thousand characters (in fact, 31 * 2^11 codes), and uses exactly two bytes per character. It is a mere dump of the internal memory representation which is natural for this subset and as such, conveys with it endianness problems.

A non-empty UCS-2 file normally begins with a so called byte order mark, having value 0xFEFF. The value 0xFFFE is not an UCS character, so if this value is seen at the beginning of a file, recode reacts by swapping all pairs of bytes. The library also properly reacts to other occurrences of 0xFEFF or 0xFFFE elsewhere than at the beginning, because concatenation of UCS-2 files should stay a simple matter, but it might trigger a diagnostic about non canonical input.

By default, when producing an UCS-2 file, recode always outputs the high order byte before the low order byte. But this could be easily overridden through the 21-Permutation surface (see Permutations). For example, the command:

recode u8..u2/21 < input > output

asks for an UTF-8 to UCS-2 conversion, with swapped byte output.

Use UCS-2 as a genuine charset. This charset is available in recode under the name ISO-10646-UCS-2. Accepted aliases are UCS-2, BMP, rune and u2.

The recode library is able to combine UCS-2 some sequences of codes into single code characters, to represent a few diacriticized characters, ligatures or diphtongs which have been included to ease mapping with other existing charsets. It is also able to explode such single code characters into the corresponding sequence of codes. The request syntax for triggering such operations is rudimentary and temporary. The combined-UCS-2 pseudo character set is a special form of UCS-2 in which known combinings have been replaced by the simpler code. Using combined-UCS-2 instead of UCS-2 in an after position of a request forces a combining step, while using combined-UCS-2 instead of UCS-2 in a before position of a request forces an exploding step. For the time being, one has to resort to advanced request syntax to achieve other effects. For example:

recode u8..co,u2..u8 < input > output

copies an UTF-8 input over output, still to be in UTF-8, yet merging combining characters into single codes whenever possible.

5.2 Universal Character Set, 4 bytes

Another surface of UCS uses exactly four bytes per character, and is a mere dump of the internal memory representation which is natural for the whole charset and as such, conveys with it endianness problems.

Use it as a genuine charset. This charset is available in recode under the name ISO-10646-UCS-4. Accepted aliases are UCS, UCS-4, ISO_10646, 10646 and u4.

5.3 Universal Transformation Format, 7 bits

UTF-7 comes from IETF rather than ISO, and is described by RFC 2152, in the MIME series. The UTF-7 encoding is meant to fit UCS-2 over channels limited to seven bits per byte. It proceeds from a mix between the spirit of Quoted-Printable and methods of Base64, adapted to Unicode contexts.

This charset is available in recode under the name UNICODE-1-1-UTF-7. Accepted aliases are UTF-7, TF-7 and u7.

5.4 Universal Transformation Format, 8 bits

Even if UTF-8 does not originally come from IETF, there is now RFC 2279 to describe it. In letters sent on 1995-01-21 and 1995-04-20, Markus Kuhn writes:

UTF-8 is an ASCII compatible multi-byte encoding of the ISO 10646 universal character set (UCS). UCS is a 31-bit superset of all other character set standards. The first 256 characters of UCS are identical to those of ISO 8859-1 (Latin-1). The UCS-2 encoding of UCS is a sequence of bigendian 16-bit words, the UCS-4 encoding is a sequence of bigendian 32-bit words. The UCS-2 subset of ISO 10646 is also known as “Unicode”. As both UCS-2 and UCS-4 require heavy modifications to traditional ASCII oriented system designs (e.g. Unix), the UTF-8 encoding has been designed for these applications.

In UTF-8, only ASCII characters are encoded using bytes below 128. All other non-ASCII characters are encoded as multi-byte sequences consisting only of bytes in the range 128-253. This avoids critical bytes like NUL and / in UTF-8 strings, which makes the UTF-8 encoding suitable for being handled by the standard C string library and being used in Unix file names. Other properties include the preserved lexical sorting order and that UTF-8 allows easy self-synchronisation of software receiving UTF-8 strings.

UTF-8 is the most common external surface of UCS, each character uses from one to six bytes, and is able to encode all 2^31 characters of the UCS. It is implemented as a charset, with the following properties:

• Strict 7-bit ASCII is completely invariant under UTF-8, and those are the only one-byte characters. UCS values and ASCII values coincide. No multi-byte characters ever contain bytes less than 128. NUL is NUL. A multi-byte character always starts with a byte of 192 or more, and is always followed by a number of bytes between 128 to 191. That means that you may read at random on disk or memory, and easily discover the start of the current, next or previous character. You can count, skip or extract characters with this only knowledge.
• If you read the first byte of a multi-byte character in binary, it contains many ‘1’ bits in successions starting with the most significant one (from the left), at least two. The length of this ‘1’ sequence equals the byte size of the character. All succeeding bytes start by ‘10’. This is a lot of redundancy, making it fairly easy to guess that a file is valid UTF-8, or to safely state that it is not.
• In a multi-byte character, if you remove all leading ‘1’ bits of the first byte of a multi-byte character, and the initial ‘10’ bits of all remaining bytes (so keeping 6 bits per byte for those), the remaining bits concatenated are the UCS value.

These properties also have a few nice consequences:

• Conversion to/from values is algorithmically simple, and reasonably speedy.
• A sequence of N bytes can hold characters needing up to 2 + 5N bits in their UCS representation. Here, N is a number between 1 and 6. So, UTF-8 is most economical when mapping ASCII (1 byte), followed by UCS-2 (1 to 3 bytes) and UCS-4 (1 to 6 bytes).
• The lexicographic sorting order of UCS strings is preserved.
• Bytes with value 254 or 255 never appear, and because of that, these are sometimes used when escape mechanisms are needed.

In some case, when little processing is done on a lot of strings, one may choose for efficiency reasons to handle UTF-8 strings directly even if variable length, as it is easy to get start of characters. Character insertion or replacement might require moving the remainder of the string in either direction. In most cases, it is faster and easier to convert from UTF-8 to UCS-2 or UCS-4 prior to processing.

This charset is available in recode under the name UTF-8. Accepted aliases are UTF-2, UTF-FSS, FSS_UTF, TF-8 and u8.

5.5 Universal Transformation Format, 16 bits

Another external surface of UCS is also variable length, each character using either two or four bytes. It is usable for the subset defined by the first million characters (17 * 2^16) of UCS.

Martin J. Dürst writes (to comp.std.internat, on 1995-03-28):

UTF-16 is another method that reserves two times 1024 codepoints in Unicode and uses them to index around one million additional characters. UTF-16 is a little bit like former multibyte codes, but quite not so, as both the first and the second 16-bit code clearly show what they are. The idea is that one million codepoints should be enough for all the rare Chinese ideograms and historical scripts that do not fit into the Base Multilingual Plane of ISO 10646 (with just about 63,000 positions available, now that 2,000 are gone).

This charset is available in recode under the name UTF-16. Accepted aliases are Unicode, TF-16 and u6.

5.6 Frequency count of characters

A device may be used to obtain a list of characters in a file, and how many times each character appears. Each count is followed by the UCS-2 value of the character and, when known, the RFC 1345 mnemonic for that character.

This charset is available in recode under the name count-characters.

This count feature has been implemented as a charset. This may change in some later version, as it would sometimes be convenient to count original bytes, instead of their UCS-2 equivalent.

5.7 Fully interpreted UCS dump

Another device may be used to get fully interpreted dumps of an UCS-2 stream of characters, with one UCS-2 character displayed on a full output line. Each line receives the RFC 1345 mnemonic for the character if it exists, the UCS-2 value of the character, and a descriptive comment for that character. As each input character produces its own output line, beware that the output file from this conversion may be much, much bigger than the input file.

This charset is available in recode under the name dump-with-names.

This dump-with-names feature has been implemented as a charset rather than a surface. This is surely debatable. The current implementation allows for dumping charsets other than UCS-2. For example, the command ‘recode l2..full < input’ implies a necessary conversion from Latin-2 to UCS-2, as dump-with-names is only connected out from UCS-2. In such cases, recode does not display the original Latin-2 codes in the dump, only the corresponding UCS-2 values. To give a simpler example, the command

echo 'Hello, world!' | recode us..dump

produces the following output:

UCS2   Mne   Description

0048   H     latin capital letter h
0065   e     latin small letter e
006C   l     latin small letter l
006C   l     latin small letter l
006F   o     latin small letter o
002C   ,     comma
0020   SP    space
0077   w     latin small letter w
006F   o     latin small letter o
0072   r     latin small letter r
006C   l     latin small letter l
0064   d     latin small letter d
0021   !     exclamation mark
000A   LF    line feed (lf)

The descriptive comment is given in English and ASCII, yet if the English description is not available but a French one is, then the French description is given instead, using Latin-1. However, if the LANGUAGE or LANG environment variable begins with the letters ‘fr’, then listing preference goes to French when both descriptions are available.

Here is another example. To get the long description of the code 237 in Latin-5 table, one may use the following command.

echo -n 237 | recode l5/d..dump

If your echo does not grok ‘-n’, use ‘echo 237\c’ instead. Here is how to see what Unicode U+03C6 means, while getting rid of the title lines.

echo -n 0x03C6 | recode u2/x2..dump | tail +3