cvt-utf8
cvt-utf8
– convert between UTF-8 and Unicode, and analyse Unicode
cvt-utf8 [flags] [hex UTF-8 bytes, U+codepoints, SGML entities]
cvt-utf8
is a tool for manipulating and analysing UTF-8 and Unicode data. Its functions include:
By default, cvt-utf8
expects to receive character data on the command line (as a mixture of UTF-8 bytes, Unicode code points and SGML numeric character entities), and it will print out a verbose analysis of the input data. If you need it to read UTF-8 from standard input or to write pure UTF-8 to standard output, you can do so using command-line options.
-i
-o
-h
In cvt-utf8
's native mode, it simply analyses input Unicode or UTF-8 data. For example, you can give a list of Unicode code points...
$ cvt-utf8 U+20ac U+31 U+30
U-000020AC E2 82 AC EURO SIGN
U-00000031 31 DIGIT ONE
U-00000030 30 DIGIT ZERO
... and cvt-utf8
gives you the UTF-8 encodings plus the character definitions.
If it's more convenient, you can specify those characters as SGML numeric entity references (for example if you're cutting and pasting out of a web page):
$ cvt-utf8 '€' '–'
U-000020AC E2 82 AC EURO SIGN
U-00002013 E2 80 93 EN DASH
Alternatively, you can supply a list of UTF-8 bytes...
$ cvt-utf8 D0 A0 D1 83 D1 81 D1 81 D0 BA D0 B8 D0 B9
U-00000420 D0 A0 CYRILLIC CAPITAL LETTER ER
U-00000443 D1 83 CYRILLIC SMALL LETTER U
U-00000441 D1 81 CYRILLIC SMALL LETTER ES
U-00000441 D1 81 CYRILLIC SMALL LETTER ES
U-0000043A D0 BA CYRILLIC SMALL LETTER KA
U-00000438 D0 B8 CYRILLIC SMALL LETTER I
U-00000439 D0 B9 CYRILLIC SMALL LETTER SHORT I
... and you get back the same output format, including the UTF-8 code points.
If you supply malformed data, cvt-utf8
will break it down for you and identify the malformed pieces and any correctly formed characters:
$ cvt-utf8 A9 FE 45 C2 80 90 0A
A9 (unexpected continuation byte)
FE (invalid UTF-8 byte)
U-00000045 45 LATIN CAPITAL LETTER E
U-00000080 C2 80 <control>
90 (unexpected continuation byte)
U-0000000A 0A <control>
If you need the UTF-8 encoding of a particular character, you can use the -o
option to cause the UTF-8 to be written to standard output:
$ cvt-utf8 -o U+20AC >> my-utf8-file.txt
If you have UTF-8 data in a file or output from another program, you can use the -i
option to have cvt-utf8
analyse it. This works particularly well if you also have my xcopy
program, which can be told to extract UTF-8 data from the X selection and write it to its standard output. With these two programs working together, if you ever have trouble identifying some text in a UTF-8-supporting web browser such as Mozilla, you can simply select the text in question, switch to a terminal window, and type
$ xcopy -u -r | cvt-utf8 -i
If the text is in Chinese, you can get at least a general idea of its meaning by using the -h
option to print the meaning of each ideograph from the Unihan database. For example, if you pass in the Chinese text meaning ‘Traditional Chinese’:
$ cvt-utf8 -h U+7E41 U+9AD4 U+4E2D U+6587
U-00007E41 E7 B9 81 <han> complicated, complex, difficult
U-00009AD4 E9 AB 94 <han> body; group, class, body, unit
U-00004E2D E4 B8 AD <han> central; center, middle; in the
midst of; hit (target); attain
U-00006587 E6 96 87 <han> literature, culture, writing
In order to print the unicode.org
official name of each character, cvt-utf8
requires a file mapping code points to names. This file is in DBM database format, for rapid lookup.
This database file is accessed using the Python anydbm
module, so its precise file name will vary depending on what flavours of DBM you have installed. The name Python knows it by is ‘unicode
’; it may actually be called ‘unicode.db
’ or something similar.
cvt-utf8
generates this DBM file itself starting from the Unicode Character Database, in the form of the file UnicodeData.txt
supplied by unicode.org
. It supports two administrative options for this purpose:
cvt-utf8 --build /path/to/UnicodeData.txt /path/to/unicode
Given a copy of UnicodeData.txt
on disk, this mode will create the DBM file and store it in a place of your choice.
cvt-utf8 --fetch-build /path/to/unicode
If you have a direct Internet connection, this will automatically download the text file from unicode.org
and process it straight into the DBM file.
There is a second DBM file, known to Python as unihan
, which is required to support the -h
option. This one is built from the Unihan Database, distributed by unicode.org
as a zip file containing a text file Unihan.txt
.
If you already have Unihan.txt
on your system, you can build cvt-utf8
's unihan
DBM file like this:
cvt-utf8 --build-unihan /path/to/Unihan.txt /path/to/unihan
Or, again, cvt-utf8
can automatically download it from unicode.org
, unpack the zip file on the fly, and write the DBM straight out:
cvt-utf8 --fetch-build-unihan /path/to/unihan
cvt-utf8
expects to find these database files in one of the following locations:
/usr/share/unicode
/usr/lib/unicode
/usr/local/share/unicode
/usr/local/lib/unicode
$HOME/share/unicode
$HOME/lib/unicode
If either of these files is not found, cvt-utf8
will still perform the rest of its functions.
cvt-utf8
is free software, distributed under the MIT licence. Type cvt-utf8 --licence
to see the full licence text.