The Erlang Token Scanner

This module contains functions for tokenizing characters into Erlang tokens.


category() = atom()

error_description() = term()

error_info() =
            {erl_anno:location(), module(), error_description()}

option() =
            return |
            return_white_spaces |
            return_comments |
            text |
            {reserved_word_fun, resword_fun()}

options() = option() | [option()]

symbol() = atom() | float() | integer() | string()

resword_fun() = fun((atom()) -> boolean())

token() =
            {category(), Anno :: erl_anno:anno(), symbol()} |
            {category(), Anno :: erl_anno:anno()}

tokens() = [token()]

tokens_result() =
            {ok, Tokens :: tokens(), EndLocation :: erl_anno:location()} |
            {eof, EndLocation :: erl_anno:location()} |
             ErrorInfo :: error_info(),
             EndLocation :: erl_anno:location()}


string(String) -> Return

string(String, StartLocation) -> Return

  • String = string()
  • Return =
        {ok, Tokens :: tokens(), EndLocation} |
        {error, ErrorInfo :: error_info(), ErrorLocation}
  • StartLocation = EndLocation = ErrorLocation = erl_anno:location()

string(String, StartLocation, Options) -> Return

Takes the list of characters String and tries to scan (tokenize) them. Returns {ok, Tokens, EndLocation}, where Tokens are the Erlang tokens from String. EndLocation is the first location after the last token.

{error, ErrorInfo, ErrorLocation} is returned if an error occurs. ErrorLocation is the first location after the erroneous token.

string(String) is equivalent to string(String, 1), and string(String, StartLocation) is equivalent to string(String, StartLocation, []).

StartLocation indicates the initial location when scanning starts. If StartLocation is a line, Anno as well as EndLocation and ErrorLocation will be lines. If StartLocation is a pair of a line and a column Anno takes the form of an opaque compound data type, and EndLocation and ErrorLocation will be pairs of a line and a column. The token annotations contain information about the column and the line where the token begins, as well as the text of the token (if the text option is given), all of which can be accessed by calling column/1, line/1, location/1, and text/1.

A token is a tuple containing information about syntactic category, the token annotations, and the actual terminal symbol. For punctuation characters (e.g. ;, |) and reserved words, the category and the symbol coincide, and the token is represented by a two-tuple. Three-tuples have one of the following forms: {atom, Info, atom()}, {char, Info, integer()}, {comment, Info, string()}, {float, Info, float()}, {integer, Info, integer()}, {var, Info, atom()}, and {white_space, Info, string()}.

The valid options are:

{reserved_word_fun, reserved_word_fun()}

A callback function that is called when the scanner has found an unquoted atom. If the function returns true, the unquoted atom itself will be the category of the token; if the function returns false, atom will be the category of the unquoted atom.


Return comment tokens.


Return white space tokens. By convention, if there is a newline character, it is always the first character of the text (there cannot be more than one newline in a white space token).


Short for [return_comments, return_white_spaces].


Include the token's text in the token annotation. The text is the part of the input corresponding to the token.

tokens(Continuation, CharSpec, StartLocation) -> Return

tokens(Continuation, CharSpec, StartLocation, Options) -> Return

  • char_spec() = string() | eof
  • return_cont()
An opaque continuation

This is the re-entrant scanner which scans characters until a dot ('.' followed by a white space) or eof has been reached. It returns:

{done, Result, LeftOverChars}

This return indicates that there is sufficient input data to get a result. Result is:

{ok, Tokens, EndLocation}

The scanning was successful. Tokens is the list of tokens including dot.

{eof, EndLocation}

End of file was encountered before any more tokens.

{error, ErrorInfo, EndLocation}

An error occurred. LeftOverChars is the remaining characters of the input data, starting from EndLocation.

{more, Continuation1}

More data is required for building a term. Continuation1 must be passed in a new call to tokens/3,4 when more data is available.

The CharSpec eof signals end of file. LeftOverChars will then take the value eof as well.

tokens(Continuation, CharSpec, StartLocation) is equivalent to tokens(Continuation, CharSpec, StartLocation, []).

See string/3 for a description of the various options.

reserved_word(Atom :: atom()) -> boolean()

Returns true if Atom is an Erlang reserved word, otherwise false.

category(Token) -> category()

Returns the category of Token.

symbol(Token) -> symbol()

Returns the symbol of Token.

column(Token) -> erl_anno:column() | undefined

Returns the column of Token's collection of annotations.

end_location(Token) -> erl_anno:location() | undefined

Returns the end location of the text of Token's collection of annotations. If there is no text, undefined is returned.

line(Token) -> erl_anno:line()

Returns the line of Token's collection of annotations.

location(Token) -> erl_anno:location()

Returns the location of Token's collection of annotations.

text(Token) -> erl_anno:text() | undefined

Returns the text of Token's collection of annotations. If there is no text, undefined is returned.

format_error(ErrorDescriptor) -> string()

Takes an ErrorDescriptor and returns a string which describes the error or warning. This function is usually called implicitly when processing an ErrorInfo structure (see below).

Error Information

The ErrorInfo mentioned above is the standard ErrorInfo structure which is returned from all IO modules. It has the following format:

{ErrorLocation, Module, ErrorDescriptor}

A string which describes the error is obtained with the following call:



The continuation of the first call to the re-entrant input functions must be []. Refer to Armstrong, Virding and Williams, 'Concurrent Programming in Erlang', Chapter 13, for a complete description of how the re-entrant input scheme works.