LAMTRACE(1)                      LAM COMMANDS                      LAMTRACE(1)


       lamtrace - Unload LAM trace data.


       lamtrace   [-hkvR]  [-mpi]  [-l  <listno>]  [-f  <#secs>]  [<filename>]
              [<nodes>] [<processes>]


       -h            Print useful information on this command.

       -k            Copy and do not remove trace data.

       -v            Be verbose.

       -R            Delete all trace data from the specified nodes.

       -l            Unload only from the given list number.

       -mpi          Unload trace data for an MPI application.

       -f <#secs>    Signal target processes to flush trace data to  the  dae-
                     mon.  Then wait <#secs> before unloading.

       <filename>    Place trace data into this file (default: def.lamtr).


       The -t option of mpirun(1) and loadgo(1) allows the application to gen-
       erate execution traces.  These traces are  first  stored  in  a  buffer
       within  each application process.  When the buffer is full and when the
       application terminates, the runtime buffer is flushed to the trace dae-
       mon  (a  structural component within the LAM daemon).  The trace daemon
       will also collect data up to a pre-compiled limit.  Beyond this  limit,
       the oldest traces in will be forgotten in favor of the newer traces.

       After  an  application  has  finished,  the  record of its execution is
       stored in the trace daemons of each node that was running the  applica-
       tion.   The  lamtrace  command can be used to retrieve these traces and
       store them in one file for display by a performance visualization tool,
       such  as  xmpi(1).  If the application was started by xmpi(1), lamtrace
       is not normally needed as the equivalent functionality is invoked  with
       a button.

       Incomplete trace data can be unloaded while the application is running.
       The output file must not exist prior to invoking lamtrace.  This  is  a
       good situation to use the -k option, which preserves the trace daemon’s
       contents after unloading.  Each reload will then get the  entire  run’s
       trace data up to the present time.

       A running process is likely to be holding the most recent trace data in
       an internal buffer.  A standard LAM signal, LAM_SIGTRACE (see doom(1)),
       causes  trace  enabled  processes to flush the internal trace buffer to
       the daemon.  The -f option tells lamtrace to send this  signal  to  all
       target  processes before unloading trace data.  A race condition devel-
       ops between the target process storing trace data to the daemon and the
       unloading  procedure.  The problem is foisted upon the user who gives a
       delay parameter after -f.

       Trace data are organized by node, process identifier and  list  number.
       A  process can store traces on any node, although the local node is the
       obvious, least intrusive choice.  The process can  identify  itself  in
       any  meaningful  way (getpid(2) is a good idea) The list number is also
       chosen by the process.  These values may be set by an instrumented  li-
       brary,  such as libmpi(3), or directly by the application with lam_rtr-
       store(2).  Unloading flexibility follows that of storing  with  the  -l
       option selecting the list number, and standard LAM command line mnemon-
       ics selecting nodes and processes.

       Dropping old traces when a pre-compiled volume limit  is  reached  only
       happens for positive list numbers.  Traces in negatively numbered lists
       will be collected until the underlying system runs out of  memory.   Do
       not use negative list numbers for high volume trace data.

       If  no  process selection is given on the command line, trace data will
       be unloaded for all processes on each specified node.

       LAM, its trace daemon and lamtrace are all unaware of  the  format  and
       meaning of traces.

       The  -R  option does not unload trace data.  It causes the target trace
       daemons to free the memory occupied by trace data in  the  given  list.
       If  all  lists are specified (no -l option), the trace daemon is effec-
       tively reset to its state after initiating LAM.

   Unloading MPI Trace Data
       A special capability, selected by the -mpi option, exists to search for
       and  unload  only  the trace data generated by an MPI application.  For
       this purpose, lamtrace is aware of the particular reserved list numbers
       that libmpi(3) uses to store traces.  It begins by searching all speci-
       fied nodes and processes (the whole LAM multicomputer,  if  nothing  is
       specified)  for  a  special  trace  generated  by  process  rank  0  in
       MPI_COMM_WORLD of an MPI application.  This special trace contains  the
       node  and  process  identifiers of all processes in that MPI_COMM_WORLD
       communicator.  lamtrace then uses the node  /  process  information  to
       collect all trace data generated by libmpi(3).

       If  multiple  world communicators exist within LAM’s trace daemons, the
       first one found is used.  Multiple worlds may be present due to  multi-
       ple concurrent applications, trace data from a previous run not removed
       (either with lamtrace or lamclean(1)), or an  application  that  spawns
       processes.  A particular world communicator can be located by providing
       precise node and process location to lamtrace.

       The -mpi option is not compatible with the -l option.


       lamtrace -v -mpi mytraces
           Unload trace data into the file "mytraces" from the first  MPI  ap-
           plication  found  in a search of the entire LAM multicomputer.  Re-
           port on important steps as they are done.

       lamtrace n30 -l 5 p21367
           Unload trace data from list 5 of process ID 21367 on node 30.   Op-
           erate silently.

       lamtrace -mpi n30 p21367
           Unload  trace  data from the MPI application world group whose pro-
           cess rank 0 has PID 21367 and is/was running on node 30.


       Since trace data can be unloaded  during  an  application’s  execution,
       there  should be a way to incrementally append to an output file.  This
       is a bit tricky with -mpi, but it can be done.


       def.lamtr     default output file


       mpirun(1), loadgo(1), lam_rtrstore(1), lamclean(1), libmpi(3), xmpi(1)

LAM 7.1.1                       September, 2004                    LAMTRACE(1)

Man(1) output converted with man2html