A base for building trace tools for distributed systems.

The Trace Tool Builder ttb is a base for building trace tools for distributed systems.

When using ttb, dbg shall not be used in parallel.


tracer() -> Result

This is equivalent to tracer(node()).

tracer(Nodes) -> Result

This is equivalent to tracer(Nodes,[]).

tracer(Nodes,Opts) -> Result

  • Result = {ok, ActivatedNodes} | {error,Reason}
  • Nodes = atom() | [atom()] | all | existing | new
  • Opts = [Opt]
  • Opt = {file,Client} | {handler, FormatHandler} | {process_info,PI}
  • Client = File | {local, File}
  • File = Filename | Wrap
  • Filename = string()
  • Wrap = {wrap,Filename} | {wrap,Filename,Size,Count}
  • FormatHandler = See format/2
  • PI = true | false

This function starts a file trace port on all given nodes and also points the system tracer for sequential tracing to the same port.

The given Filename will be prefixed with the node name. Default Filename is "ttb".

File={wrap,Filename,Size,Count} can be used if the size of the trace logs must be limited. Default values are Size=128*1024 and Count=8.

When tracing diskless nodes, ttb must be started from an external "trace control node" with disk access, and Client must be {local, File}. All trace information is then sent to the trace control node where it is written to file.

The process_info option indicates if process information should be collected. If PI = true (which is default), each process identifier Pid is replaced by a tuple {Pid,ProcessInfo,Node}, where ProcessInfo is the process' registered name its globally registered name, or its initial function. It is possible to turn off this functionality by setting PI = false.

p(Procs,Flags) -> Return

  • Return = {ok,[{Procs,MatchDesc}]}
  • Procs = Process | [Process] | all | new | existing
  • Process = pid() | atom() | {global,atom()}
  • Flags = Flag | [Flag]

This function sets the given trace flags on the given processes.

Please turn to the Reference manual for module dbg for details about the possible trace flags. The parameter MatchDesc is the same as returned from dbg:p/2

Processes can be given as registered names, globally registered names or process identifiers. If a registered name is given, the flags are set on processes with this name on all active nodes.

These functions should be used in combination with the call trace flag for setting and clearing trace patterns. When the call trace flag is set on a process, function calls will be traced on that process if a trace pattern has been set for the called function. Trace patterns specifies how to trace a function by using match specifications. Match specifications are described in the User's Guide for the erlang runtime system erts.

These functions are equivalent to the corresponding functions in dbg, but all calls are stored in the history. The history buffer makes it easy to create config files so that the same trace environment can be setup several times, e.g. if you want to compare two test runs. It also reduces the amount of typing when using ttb from the erlang shell.

Set trace pattern on global function calls
Set trace pattern on local and global function calls
Clear trace pattern on local and global function calls
Clear trace pattern on local function calls
Clear trace pattern on global function calls

list_history() -> History

  • History = [{N,Func,Args}]

All calls to ttb is stored in the history. This function returns the current content of the history. Any entry can be re-executed with run_history/1 or stored in a config file with write_config/2/3.

run_history(N) -> ok | {error, Reason}

  • N = integer() | [integer()]

Executes the given entry or entries from the history list. History can be listed with list_history/0.


Equivalent to write_config(ConfigFile,Config,[]).

write_config(ConfigFile,Config,Opt) -> ok | {error,Reason}

  • ConfigFile = string()
  • Config = all | [integer()] | [{Mod,Func,Args}]
  • Mod = atom()
  • Func = atom()
  • Args = [term()]
  • Opt = [] | [append]

This function creates or extends a config file which can be used for restoring a specific configuration later.

The content of the config file can either be fetched from the history or given directly as a list of {Mod,Func,Args}.

If the complete history is to be stored in the config file Config should be all. If only a selected number of entries from the history should be stored, Config should be a list of integers pointing out the entries to be stored.

If Opt is not given or if it is [], ConfigFile is deleted and a new file is created. If Opt = [append], ConfigFile will not be deleted. The new information will be appended at the end of the file.

run_config(ConfigFile) -> ok | {error,Reason}

  • ConfigFile = string()

Executes all entries in the given config file.

run_config(ConfigFile,NumList) -> ok | {error,Reason}

  • ConfigFile = string()
  • NumList = [integer()]

Executes selected entries from the given config file. NumList is a list of integers pointing out the entries to be executed.

The content of a config file can be listed with list_config/1.

list_config(ConfigFile) -> Config | {error,Reason}

  • ConfigFile = string()
  • Config = [{N,Func,Args}]

Lists all entries in the given config file.

write_trace_info(Key,Info) -> ok

  • Key = term()
  • Info = Data | fun() -> Data
  • Data = term()

The .ti file contains {Key,ValueList} tuples. This function adds Data to the ValueList associated with Key. All information written with this function will be included in the call to the format handler.

seq_trigger_ms() -> MatchSpec

Equivalent to seq_trigger_ms(all)

seq_trigger_ms(Flags) -> MatchSpec

  • MatchSpec = match_spec()
  • Flags = all | SeqTraceFlag | [SeqTraceFlag]
  • SeqTraceFlag = atom()

A match specification can turn on or off sequential tracing. This function returns a match specification which turns on sequential tracing with the given Flags.

This match specification can be given as the last argument to tp or tpl. The activated Item will then become a trigger for sequential tracing. This means that if the item is called on a process with the call trace flag set, the process will be "contaminated" with the seq_trace token.

If Flags = all, all possible flags are set.

Please turn to the reference manual for the seq_trace module in the kernel application to see the possible values for SeqTraceFlag. For a description of the match_spec() syntax, please turn to the User's guide for the runtime system (erts). The chapter Match Specification in Erlang explains the general match specification "language".


The system tracer for sequential tracing is automatically initiated by ttb when a trace port is started with ttb:tracer/0/1/2.

Example of how to use the seq_trigger_ms/0/1 function:

(tiger@durin)5> ttb:tracer().
(tiger@durin)6> ttb:p(all,call).
(tiger@durin)7> ttb:tp(mod,func,ttb:seq_trigger_ms()).

Whenever mod:func(...) is called after this, the seq_trace token will be set on the executing process.


Equivalent to stop([]).

stop(Opts) -> stopped

  • Opts = [Opt]
  • Opt = fetch | format

Stops tracing on all nodes.

The fetch option indicates that trace logs shall be collected from all nodes after tracing is stopped. This option is useful if nodes on remote machines are traced. Logs and trace information files are then sent to the trace control node and stored in a directory named ttb_upload-Timestamp, where Timestamp is on the form yyyymmdd-hhmmss. Even logs from nodes on the same machine as the trace control node are moved to this directory.

The format option indicates that the trace logs shall be formatted after tracing is stopped. Note that this option also implies the fetch option, i.e. logs are collected in a new directory on the trace control node before formatting. All logs in the directory will be merged.


Same as format(File,[]).

format(File,Options) -> ok | {error, Reason}

  • File = string() | [string()]
  • This can be the name of a binary log, a list of such logs or the name of a directory containing one or more binary logs.
  • Options = [Opt]
  • Opt = {out,Out} | {handler,FormatHandler}
  • Out = standard_io | string()
  • FormatHandler = {Function, InitialState} | et
  • Function = fun(Fd,Trace,TraceInfo,State) -> State
  • Fd = standard_io | FileDescriptor
  • This is the file descriptor of the destination file Out
  • Trace = tuple()
  • This is the trace message. Please turn to the Reference manual for the erlangmodule for details.
  • TraceInfo = [{Key,ValueList}]
  • This includes the keys flags, clientand node, and if handleris given as option to the tracer function, this is also included. In addition all information written with the write_trace_info/2function is included.

Reads the given binary trace log(s). If a directory or a list of logs is given and the timestamp flag was set during tracing, the trace messages from the different logs are merged according to the timestamps.

If FormatHandler = {Function,InitialState}, Function will be called for each trace message. If FormatHandler = et, et_viewer in the Event Tracer application (et) is used for presenting the trace log graphically. ttb provides a few different filters which can be selected from the Filter menu in the et_viewer. If FormatHandler is not given, a default handler is used which presents each trace message as a line of text.

If Out is given, FormatHandler gets the filedescriptor to Out as the first parameter.

Out is ignored if FormatHandler = et.

Wrap logs can be formatted one by one or all in one go. To format one of the wrap logs in a set, give the exact name of the file. To format the whole set of wrap logs, give the name with '*' instead of the wrap count. See examples in the ttb User's Guide.

View Functions