erl_scan
The Erlang token scanner.
This module contains functions for tokenizing (scanning) characters into Erlang tokens.
Types
category() = atom()
error_description() = term()
error_info() =
{erl_anno:location(), module(), error_description()}
option() =
return |
return_white_spaces |
return_comments |
text |
{reserved_word_fun, resword_fun()}
symbol() = atom() | float() | integer() | string()
resword_fun() = fun((atom()) -> boolean())
token() =
{category(), Anno :: erl_anno:anno(), symbol()} |
{category(), Anno :: erl_anno:anno()}
tokens() = [token()]
tokens_result() =
{ok, Tokens :: tokens(), EndLocation :: erl_anno:location()} |
{eof, EndLocation :: erl_anno:location()} |
{error,
ErrorInfo :: error_info(),
EndLocation :: erl_anno:location()}
Functions
column(Token) -> erl_anno:column() | undefined
Token = token()
Returns the column of
's
collection of annotations.
end_location(Token) -> erl_anno:location() | undefined
Token = token()
Returns the end location of the text of
's collection of annotations. If
there is no text, undefined
is returned.
format_error(ErrorDescriptor) -> string()
ErrorDescriptor = error_description()
Uses an
and returns a string
that describes the error or warning. This function is usually
called implicitly when an ErrorInfo
structure is
processed (see section
Error Information).
line(Token) -> erl_anno:line()
Token = token()
Returns the line of
's collection
of annotations.
location(Token) -> erl_anno:location()
Token = token()
Returns the location of
's
collection of annotations.
reserved_word(Atom :: atom()) -> boolean()
Returns true
if
is an
Erlang reserved word, otherwise false
.
string(String) -> Return
String = string()
Return =
{ok, Tokens :: tokens(), EndLocation} |
{error, ErrorInfo :: error_info(), ErrorLocation}EndLocation = ErrorLocation = erl_anno:location()
string(String, StartLocation) -> Return
String = string()
Return =
{ok, Tokens :: tokens(), EndLocation} |
{error, ErrorInfo :: error_info(), ErrorLocation}StartLocation = EndLocation = ErrorLocation = erl_anno:location()
string(String, StartLocation, Options) -> Return
String = string()
Options = options()
Return =
{ok, Tokens :: tokens(), EndLocation} |
{error, ErrorInfo :: error_info(), ErrorLocation}StartLocation = EndLocation = ErrorLocation = erl_anno:location()
Takes the list of characters
and tries to
scan (tokenize) them. Returns one of the following:
{ok, Tokens , EndLocation }
are the Erlang tokens from
.
is the first location after the last token.
{error, ErrorInfo ,
ErrorLocation }
An error occurred.
is the
first location after the erroneous token.
string(
is equivalent to
string(
, and
string(
is equivalent to
string(
.
indicates the initial location
when scanning starts. If
is a line,
Anno
,
, and
are lines. If
is a pair of a line and a column,
Anno
takes the form of an opaque compound
data type, and
and
are pairs of a line and a column. The token
annotations contain information about the column and the
line where the token begins, as well as the text of the
token (if option text
is specified), all of which can
be accessed by calling
column/1
,
line/1
,
location/1
, and
text/1
.
A token is a tuple containing information about
syntactic category, the token annotations, and the
terminal symbol. For punctuation characters (such as ;
and
|
) and reserved words, the category and the symbol
coincide, and the token is represented by a two-tuple.
Three-tuples have one of the following forms:
{atom, Anno, atom()}
{char, Anno, char()}
{comment, Anno, string()}
{float, Anno, float()}
{integer, Anno, integer()}
{var, Anno, atom()}
{white_space, Anno, string()}
Valid options:
{reserved_word_fun, reserved_word_fun()}
A callback function that is called when the scanner
has found an unquoted atom. If the function returns
true
, the unquoted atom itself becomes the category
of the token. If the function returns false
,
atom
becomes the category of the unquoted atom.
return_comments
Return comment tokens.
return_white_spaces
Return white space tokens. By convention, a newline character, if present, is always the first character of the text (there cannot be more than one newline in a white space token).
return
Short for [return_comments, return_white_spaces]
.
text
Include the token text in the token annotation. The text is the part of the input corresponding to the token.
text(Token) -> erl_anno:text() | undefined
Token = token()
Returns the text of
's collection
of annotations. If there is no text, undefined
is
returned.
tokens(Continuation, CharSpec, StartLocation) -> Return
Continuation = return_cont() | []
CharSpec = char_spec()
StartLocation = erl_anno:location()
Return =
{done,
Result :: tokens_result(),
LeftOverChars :: char_spec()} |
{more, Continuation1 :: return_cont()}
tokens(Continuation, CharSpec, StartLocation, Options) -> Return
Continuation = return_cont() | []
CharSpec = char_spec()
StartLocation = erl_anno:location()
Options = options()
Return =
{done,
Result :: tokens_result(),
LeftOverChars :: char_spec()} |
{more, Continuation1 :: return_cont()}
char_spec() = string() | eof
return_cont()
This is the re-entrant scanner, which scans characters until
either a dot ('.' followed by a white space) or
eof
is reached. It returns:
{done, Result , LeftOverChars }
Indicates that there is sufficient input
data to get a result.
is:
{ok, Tokens, EndLocation}
The scanning was successful. Tokens
is the list of tokens including dot.
{eof, EndLocation}
End of file was encountered before any more tokens.
{error, ErrorInfo, EndLocation}
An error occurred.
is the remaining characters of the input data,
starting from EndLocation
.
{more, Continuation1 }
More data is required for building a term.
must be passed in a new call to
tokens/3,4
when more data is available.
The eof
signals end of file.
then takes the value eof
as well.
tokens(
is equivalent to
tokens(
.
For a description of the options, see
string/3
.
Error Information
ErrorInfo
is the standard ErrorInfo
structure that is
returned from all I/O modules. The format is as follows:
{ErrorLocation, Module, ErrorDescriptor}
A string describing the error is obtained with the following call:
Module:format_error(ErrorDescriptor)
Notes
The continuation of the first call to the re-entrant input
functions must be []
. For a complete description of how the
re-entrant input scheme works, see Armstrong, Virding and
Williams: 'Concurrent Programming in Erlang', Chapter 13.