erl_scan
The Erlang Token Scanner
This module contains functions for tokenizing characters into Erlang tokens.
DATA TYPES
category() = atom() column() = integer() > 0 line() = integer() location() = line() | {line(), column()} reserved_word_fun() -> fun(atom()) -> bool() set_attribute_fun() -> fun(term()) -> term() symbol() = atom() | float() | integer() | string() token() = {category(), attributes()} | {category(), attributes(), symbol()} attributes() = line() | list() | tuple()
Functions
string(String) -> Return
string(String, StartLocation) -> Return
string(String, StartLocation, Options) -> Return
String = string()
Return = {ok, Tokens, EndLocation} | Error
Tokens = [token()]
Error = {error, ErrorInfo, EndLocation}
StartLocation = EndLocation = location()
Options = Option | [Option]
Option = {reserved_word_fun,reserved_word_fun()} | return_comments | return_white_spaces | return | text
Takes the list of characters String
and tries to
scan (tokenize) them. Returns {ok, Tokens, EndLocation}
,
where Tokens
are the Erlang tokens from
String
. EndLocation
is the first location
after the last token.
{error, ErrorInfo, EndLocation}
is returned if an
error occurs. EndLocation
is the first location after
the erroneous token.
string(String)
is equivalent to
string(String, 1)
, and string(String,
StartLocation)
is equivalent to string(String,
StartLocation, [])
.
StartLocation
indicates the initial location when
scanning starts. If StartLocation
is a line
attributes()
as well as EndLocation
and
ErrorLocation
will be lines. If
StartLocation
is a pair of a line and a column
attributes()
takes the form of an opaque compound
data type, and EndLocation
and ErrorLocation
will be pairs of a line and a column. The token
attributes contain information about the column and the
line where the token begins, as well as the text of the
token (if the text
option is given), all of which can
be accessed by calling token_info/1,2 or attributes_info/1,2.
A token is a tuple containing information about
syntactic category, the token attributes, and the actual
terminal symbol. For punctuation characters (e.g. ;
,
|
) and reserved words, the category and the symbol
coincide, and the token is represented by a two-tuple.
Three-tuples have one of the following forms: {atom,
Info, atom()}
,
{char, Info, integer()}
, {comment, Info,
string()}
, {float, Info, float()}
, {integer,
Info, integer()}
, {var, Info, atom()}
,
and {white_space, Info, string()}
.
The valid options are:
{reserved_word_fun, reserved_word_fun()}
A callback function that is called when the scanner has found an unquoted atom. If the function returns
true
, the unquoted atom itself will be the category of the token; if the function returnsfalse
,atom
will be the category of the unquoted atom.return_comments
Return comment tokens.
return_white_spaces
Return white space tokens. By convention, if there is a newline character, it is always the first character of the text (there cannot be more than one newline in a white space token).
return
Short for
[return_comments, return_white_spaces]
.text
Include the token's text in the token attributes. The text is the part of the input corresponding to the token.
tokens(Continuation, CharSpec, StartLocation) -> Return
tokens(Continuation, CharSpec, StartLocation, Options) -> Return
Continuation = [] | Continuation1
Return = {done, Result, LeftOverChars} | {more, Continuation1}
LeftOverChars = CharSpec
CharSpec = string() | eof
Continuation1 = tuple()
Result = {ok, Tokens, EndLocation} | {eof, EndLocation} | Error
Tokens = [token()]
Error = {error, ErrorInfo, EndLocation}
StartLocation = EndLocation = location()
Options = Option | [Option]
Option = {reserved_word_fun,reserved_word_fun()} | return_comments | return_white_spaces | return | text
This is the re-entrant scanner which scans characters until
a dot ('.' followed by a white space) or
eof
has been reached. It returns:
{done, Result, LeftOverChars}
-
This return indicates that there is sufficient input data to get a result.
Result
is:{ok, Tokens, EndLocation}
-
The scanning was successful.
Tokens
is the list of tokens including dot. {eof, EndLocation}
-
End of file was encountered before any more tokens.
{error, ErrorInfo, EndLocation}
-
An error occurred.
LeftOverChars
is the remaining characters of the input data, starting fromEndLocation
.
{more, Continuation1}
-
More data is required for building a term.
Continuation1
must be passed in a new call totokens/3,4
when more data is available.
The CharSpec
eof
signals end of file.
LeftOverChars
will then take the value eof
as
well.
tokens(Continuation, CharSpec, StartLocation)
is
equivalent to tokens(Continuation, CharSpec,
StartLocation, [])
.
See string/3 for a description of the various options.
reserved_word(Atom) -> bool()
Atom = atom()
Returns true
if Atom
is an Erlang reserved
word, otherwise false
.
token_info(Token) -> TokenInfo
Token = token()
TokenInfo = [TokenInfoTuple]
TokenInfoTuple = {TokenItem, Info}
TokenItem = atom()
Info = term()
Returns a list containing information about the token
Token
. The order of the TokenInfoTuple
s is not
defined. The following TokenItem
s are returned:
category
, column
, length
,
line
, symbol
, and text
. See token_info/2 for
information about specific
TokenInfoTuple
s.
Note that if token_info(Token, TokenItem)
returns
undefined
for some TokenItem
in the list above, the
item is not included in TokenInfo
.
token_info(Token, TokenItemSpec) -> TokenInfo
Token = token()
TokenItemSpec = TokenItem | [TokenItem]
TokenInfo = TokenInfoTuple | undefined | [TokenInfoTuple]
TokenInfoTuple = {TokenItem, Info}
TokenItem = atom()
Info = term()
Returns a list containing information about the token
Token
. If TokenItemSpec
is a single
TokenItem
, the returned value is the corresponding
TokenInfoTuple
, or undefined
if the
TokenItem
has no value. If TokenItemSpec
is a
list of
TokenItem
, the result is a list of
TokenInfoTuple
. The TokenInfoTuple
s will
appear with the corresponding
TokenItem
s in the same order as the TokenItem
s
appeared in the list of TokenItem
s. TokenItem
s
with no value are not included in the list of
TokenInfoTuple
.
The following TokenInfoTuple
s with corresponding
TokenItem
s are valid:
{category, category()}
The category of the token.
{column, column()}
The column where the token begins.
{length, integer() > 0}
The length of the token's text.
{line, line()}
The line where the token begins.
{location, location()}
The line and column where the token begins, or just the line if the column unknown.
{symbol, symbol()}
The token's symbol.
{text, string()}
The token's text.
attributes_info(Attributes) -> AttributesInfo
Attributes = attributes()
AttributesInfo = [AttributeInfoTuple]
AttributeInfoTuple = {AttributeItem, Info}
AttributeItem = atom()
Info = term()
Returns a list containing information about the token
attributes Attributes
. The order of the
AttributeInfoTuple
s is not defined. The following
AttributeItem
s are returned:
column
, length
, line
, and text
.
See attributes_info/2 for
information about specific
AttributeInfoTuple
s.
Note that if attributes_info(Token, AttributeItem)
returns undefined
for some AttributeItem
in
the list above, the item is not included in
AttributesInfo
.
attributes_info(Attributes, AttributeItemSpec) -> AttributesInfo
Attributes = attributes()
AttributeItemSpec = AttributeItem | [AttributeItem]
AttributesInfo = AttributeInfoTuple | undefined | [AttributeInfoTuple]
AttributeInfoTuple = {AttributeItem, Info}
AttributeItem = atom()
Info = term()
Returns a list containing information about the token
attributes Attributes
. If AttributeItemSpec
is
a single AttributeItem
, the returned value is the
corresponding AttributeInfoTuple
, or undefined
if the AttributeItem
has no value. If
AttributeItemSpec
is a list of
AttributeItem
, the result is a list of
AttributeInfoTuple
. The AttributeInfoTuple
s
will appear with the corresponding AttributeItem
s in
the same order as the AttributeItem
s appeared in the
list of AttributeItem
s. AttributeItem
s with no
value are not included in the list of
AttributeInfoTuple
.
The following AttributeInfoTuple
s with corresponding
AttributeItem
s are valid:
{column, column()}
The column where the token begins.
{length, integer() > 0}
The length of the token's text.
{line, line()}
The line where the token begins.
{location, location()}
The line and column where the token begins, or just the line if the column unknown.
{text, string()}
The token's text.
set_attribute(AttributeItem, Attributes, SetAttributeFun) -> AttributesInfo
AttributeItem = line
Attributes = attributes()
SetAttributeFun = set_attribute_fun()
Sets the value of the line
attribute of the token
attributes Attributes
.
The SetAttributeFun
is called with the value of
the line
attribute, and is to return the new value of
the line
attribute.
format_error(ErrorDescriptor) -> string()
ErrorDescriptor = errordesc()
Takes an ErrorDescriptor
and returns a string which
describes the error or warning. This function is usually
called implicitly when processing an ErrorInfo
structure (see below).
Error Information
The ErrorInfo
mentioned above is the standard
ErrorInfo
structure which is returned from all IO
modules. It has the following format:
{ErrorLocation, Module, ErrorDescriptor}
A string which describes the error is obtained with the following call:
Module:format_error(ErrorDescriptor)
Notes
The continuation of the first call to the re-entrant input
functions must be []
. Refer to Armstrong, Virding and
Williams, 'Concurrent Programming in Erlang', Chapter 13, for a
complete description of how the re-entrant input scheme works.