unicode

Functions for converting Unicode characters

This module contains functions for converting between different character representations. Basically it converts between iso-latin-1 characters and Unicode ditto, but it can also convert between different Unicode encodings (like UTF-8, UTF-16 and UTF-32).

The default Unicode encoding in Erlang is in binaries UTF-8, which is also the format in which built in functions and libraries in OTP expect to find binary Unicode data. In lists, Unicode data is encoded as integers, each integer representing one character and encoded simply as the Unicode codepoint for the character.

Other Unicode encodings than integers representing codepoints or UTF-8 in binaries are referred to as "external encodings". The iso-latin-1 encoding is in binaries and lists referred to as latin1-encoding.

It is recommended to only use external encodings for communication with external entities where this is required. When working inside the Erlang/OTP environment, it is recommended to keep binaries in UTF-8 when representing Unicode characters. Latin1 encoding is supported both for backward compatibility and for communication with external entities not supporting Unicode character sets.

DATA TYPES

unicode_binary() = binary() with characters encoded in UTF-8 coding standard
unicode_char() = integer() representing valid unicode codepoint

chardata() = charlist() | unicode_binary()

charlist() = [unicode_char() | unicode_binary() | charlist()]
  a unicode_binary is allowed as the tail of the list

external_unicode_binary() = binary() with characters coded in a user specified Unicode encoding other than UTF-8 (UTF-16 or UTF-32)

external_chardata() = external_charlist() | external_unicode_binary()

external_charlist() = [unicode_char() | external_unicode_binary() | external_charlist()]
  a external_unicode_binary is allowed as the tail of the list

latin1_binary() = binary() with characters coded in iso-latin-1
latin1_char() = integer() representing valid latin1 character (0-255)

latin1_chardata() = latin1_charlist() | latin1_binary()

latin1_charlist() = [latin1_char() | latin1_binary() | latin1_charlist()]
  a latin1_binary is allowed as the tail of the list

Functions

erlang:characters_to_list(Data, Encoding) -> list() | {error, list(), RestData} | {incomplete, list(), binary()}

Data = ListData | binary()
RestData = ListData | binary()
ListData = [ int() | binary() ] (binary allowed as tail of list)
Encoding = unicode | latin1

This function converts a possibly deep list of integers and binaries into a list of integers representing unicode characters. The binaries in the input may have characters encoded as latin1 (0 - 255, one character per byte), in which case the Encoding parameter should be given as latin1, or have characters encoded as UTF-8, in which case the Encoding should be given as unicode. Only when the Encoding is unicode, integers in the list are allowed to be grater than 255.

If Encoding is latin1, the Data parameter corresponds to the iodata() type, but for unicode, the Data parameter can contain integers greater than 255 (unicode characters beyond the iso-latin-1 range), which would make it invalid as iodata().

The purpose of the function is mainly to be able to convert combinations of unicode characters into a pure unicode string in list representation for further processing. For writing the data to an external entity, the reverse function erlang:characters_to_utf8/2 comes in handy.

If for some reason, the data cannot be converted, either because of illegal unicode/latin1 characters in the list, or because of invalid UTF-8 encoding in any binaries, an error tuple is returned. The error tuple contains the tag error, a list representing the characters that could be converted before the error occured and a representation of the characters including and after the offending integer/bytes. The last part is mostly for debugging as it still constitutes a possibly deep and/or mixed list, not necessarily of the same depth as the original data. The error occurs when traversing the list and whatever's left to decode is simply returned as is.

However, if the input Data is a pure binary, the third part of the error tuple is guaranteed to be a binary as well.

Errors occur for the following reasons:

Integers out of range - If Encoding is latin1, an error occurs whenever an integer greater than 255 is found in the lists. If Encoding is unicode, error occurs whenever an integer greater than 16#10FFFF (the maximum unicode character) or in the range 16#D800 to 16#DFFF (invalid unicode range) is found. UTF-8 encoding incorrect - If Encoding is unicode, the bytes in any binaries have to be valid UTF-8. Errors can occur for various reasons, including "pure" decoding errors (like the upper bits of the bytes being wrong), the bytes are decoded to a too large number, the bytes are decoded to a code-point in the invalid unicode range or encoding is "overlong", meaning that a number should have been encoded in fewer bytes. The case of a truncated UTF-8 is handled specially, see the paragraph about incomplete binaries below. If Encoding is latin1, binaries are always valid as long as they contain whole bytes, as each byte falls into the valid iso-latin-1 range.

A special type of error is when no actual invalid integers or bytes are found, but a trailing binary() consists of too few bytes to decode the last character. This error might occur if bytes are read from a file in chunks or binaries in other ways are split on non UTF-8 boundaries. In this case an incomplete tuple is returned instead of the error tuple. It consists of the same parts as the error tuple, but the tag is incomplete instead of error and the last element is always guaranteed to be a binary consisting of the first part of a (so far) valid UTF-8 character.

If one UTF-8 characters is split over two consecutive binaries in the Data, the conversion succeeds. This means that a character can be decoded from a range of binaries as long as the whole range is given as input without errors occuring. Example:


     decode_data(Data) ->
         case erlang:characters_to_list(Data,unicode) of
             {inclomplete,Encoded, Rest} ->
	           More = get_some_more_data(),
		   Encoded ++ decode_data([Rest, More]);
	     {error,Encoded,Rest} ->
	           handle_error(Encoded,Rest);
             List ->
	           List
         end.

Bit-strings that are not whole bytes are however not allowed, so a UTF-8 character has to be split along 8-bit boundaries to ever be decoded.

If any parameters are of the wrong type, the list structure is invalid (a number as tail) or the binaries does not contain whole bytes (bit-strings), a badarg exception is thrown.

erlang:characters_to_utf8(Data, Encoding) -> binary() | {error, binary(), RestData} | {incomplete, binary(), binary()}

Data = ListData | binary()
RestData = ListData | binary()
ListData = [ int() | binary() ] (binary allowed as tail of list)
Encoding = unicode | latin1

This function behaves as erlang:characters_to_list/2, but produces an UTF-8 binary instead of a unicode list. Note that even if Encoding is given as latin1, the output is UTF-8. The Encoding defines how input is to be interpreted, not what output is generated. To convert a possibly deep list of iso-latin-1 characters to a iso-latin-1 binary, use iolist_to_binary/1.

Errors and exceptions occur as in erlang:characters_to_list/2, but of course the second element in the error or incomplete tuple will be a binary() and not a list().

View Functions

characters_to_list/2
characters_to_utf8/2