=============================================================================
                               T A P E / P V M

   T R A C E   T O O L  F O R  P E R F O R M A N C E  E V A L U A T I O N 
   ~   ~                       ~                      ~
                                 PVM VERSION

                                 Eric Maillet
                         LMC-IMAG, Grenoble, France
                            Eric.Maillet@imag.fr
==============================================================================

				    Note:

    Installation instructions can be found in INSTALL.

    For a detailed description on how to use TAPE/PVM refer to the 
    postscript user's guide, available in the same location you got
    this package from. This README is only a subset of the information
    provided in the user-guide. So if you read the user guide you won't
    need to read this README file.



			      GENERAL OVERVIEW
			      ----------------

   TAPE/PVM is a tool to generate traces on PVM applications for post-mortem
performance analysis i.e. discrete event simulation and visualization. It
only comprises the tool to generate the traces, not the post-mortem analysis
tools.  Indeed, we believe that trace generation and post-mortem analysis of
traces are two different research areas, each with its own specific problems.
The main problem of trace analysis is the design of an appropriate model and
a simulator based on that model. The simulator takes a trace file (set of
events) as input and reconstructs the successive global states of the system
on which the traces were generated. Such a simulator can be coupled to a
visualization tool to give a causally coherent global view of the system
under study. However, the simulation is only as accurate as its input - the
trace file. Such a trace file has to be representative of what really happened
in the parallel system under study. Thus, the main problem in designing a
tracing tool is to guarantee the representative quality of the generated
traces. In the design of TAPE/PVM we particularly focused on the following
two points:

   a) precise, causally coherent event dating,
   b) minimal perturbation of parallel programs by the tracing tool.
   
   Some existing tracing tools for PVM focus on user interfaces rather than
on the representative quality of the generated traces. The event dating
mechanism used in XAB [ref], for instance, is completely unrealistic. Another
tool, XPVM, is a graphical console and monitor for PVM. It uses the event
collection mechanism integrated in PVM V3.3.0 or later. Events are routed to
XPVM by the PVM kernel during run-time of the instrumented application. Thus,
XPVM can update its views in real time. XPVM can also be used for post-mortem
trace analysis using the events of previous executions saved into a file.
However, whatever the mode in which XPVM is used, real-time or post-mortem,
its traces represent potentially perturbed applications due to on-line event
message routing. These messages increase the load of the network which can
infer a change in behaviour of the observed application (in fact, many
parallel applications are non deterministic). In addition, the tracing
mechanism of the PVM kernel relies on a globally synchronized system clock.
Not many systems have a global time reference which is sufficiently accurate
to avoid dating anomalies.

   In TAPE/PVM we use a non-intrusive, statistical method which
estimates a precise global time reference [ref]. Dated events are causally
coherent.  However, the estimated global time is only available at the end of
the instrumented application which prohibits on-line dating.  This is not a
drawback because TAPE/PVM is intended for post-mortem trace analysis only.
In addition to this, at generation, an event is not routed to a central
collector task, like in XPVM, in order to avoid additional network load.
Instead the events are stored in local event buffers, which are flushed to
local event files. The collection of events into a single file is only done
at the end of the user's application to avoid interfering with it.

   The problem of perturbation of parallel applications due to the presence
of a tracing tool is a difficult one. Our approach is similar to that adopted
in the AIMS environment [ref]. We believe that, although intrusion can be
reduced by careful implementation of the tracing tool, it can not be
eliminated. The main causes of intrusion are the flushing of local event
buffers, the accumulation of the delays of each individual event generation,
as well as the additional messages exchanged by the tracing tool.  To limit
the intrusion due to TAPE/PVM we use the following techniques:

   b1) On-line compacting of events. This allows a gain of about 50% with
respect to a non-compacted text representation of events. The number of
buffer flushes is signficantly reduced and so is the perturbation of the
application.

   b2) The number of messages exchanged by TAPE/PVM is reduced to a minimum.
Only events like pvm_addhost and pvm_kill which change the configuration of
the parallel virtual machine need such additional messages.

   b3) The additional tasks used by TAPE/PVM (for global control, for clock
synchronization) are not active while the instrumented user application is
running.

   In order to facilitate subsequent intrusion compensation, TAPE/PVM
generates special events on buffer flush which contain the delay inferred.
The complexity of the event storage routine used in TAPE/PVM only depends on
the length of the stored events (compacting included). Thus, it is possible
to account for the accumulation of event storage overhead knowing the length
of the stored events.  We have already experimented such a compensation
technique on a CSP programming model (Transputer Network).

   The trace format output by TAPE/PVM is close to the PICL format [ref]. We
provide a tool (tape2picl) to transform the traces into the PICL format so
that they can be visualized with Paragraph [ref]. A special feature of
tape2picl is that it models the overhead due to buffer flushes by the
"overhead" state.  Thus, with Paragraph, the overhead due to buffer flushes
is clearly outlined on the "Task Gantt Chart" so that users can study the
intrusion by comparing different executions using different buffer sizes
(which can be parameterized in TAPE/PVM).

   Finally, to allow users to quickly find the statement in their source code
which generated a particular event, TAPE/PVM's events contain the line number
of that statement and the identifier of the source code file. In fact, the
user's source code is instrumented by TAPE/PVM's preprocessor (tapepp) which
knows the name of the file it processes and the current line number. Each
time a probe is inserted into the user's code (at a call of a PVM library
function, for instance) the information about file name and line number is
given to that probe (in a way similar to AIMS). Thus, a visualization tool
based on TAPE/PVM traces can feature source code click-back.


				RELATED WORK
				------------

   Up till now, TAPE/PVM can only be used with the visualization tool
Paragraph. However, another team of our departement of performance analysis
at LMC-IMAG is working on a simulator coupled with a visualization tool
(SCOPE) using the NeXT Step user interface, which has proven to be very
flexible due to its object oriented design. The visualization tool features
3D displays of task graphs outlining the successive states of the different
tasks and communication channels. Another view displays a 3D Kiviat. At
present, the simulator of SCOPE only supports the CSP communication model and
can be used with traces taken on Transputer networks (a simpler version of
TAPE is also available for Transputer networks). However, it will be adapted
soon to support PVM's communication model - it will then directly read
TAPE/PVM traces.


		       SUMMARY OF TAPE/PVM's FEATURES
		       ------------------------------

   - trace of events at user application level (pvm library calls) through
     function derouting
   - preprocessor to instrument user source code (C or Fortran) automatically 
     (instrumented source code has to be recompiled)
   - user defined events (like printf)
   - an event contains the line and file number of the instruction which
     generated the event (source code feed-back)
   - selective tracing through the use of event masks
   - precise, causally coherent global time reference
   - on line event compacting (gain up to 50% with respect to text
     storage) to limit event buffer flushes
   - limits perturbation of user applications
   - can generate PICL traces for use with Paragraph


				  ACKNOWLEDGE
				  -----------

   I owe much to L. Azema and M. Collet, two students at Ensimag, Grenoble,
France, who did much of the implementation of TAPE/PVM and also made
interesting suggestions as to TAPE/PVM's overall design. I also express my
thanks to Pr. B. Plateau who supervises the performance evaluation team at
LMC-IMAG.
