Newsgroups: comp.parallel.mpi
From: wcs@win233.nas.nasa.gov (William C. Saphir)
Subject: Re: Reducing overhead
Organization: Numerical Aerodynamic Simulation, NASA Ames
Date: 06 Jul 1995 22:08:08 GMT
Message-ID: <WCS.95Jul6150808@win233.nas.nasa.gov>



> 	I'm trying to find a way to cut down the communication
> overhead. Doing a standard send and receive, MPI executes three copy
> operations, from the send process to the MPI buffers, from there to
> the MPI buffers on the receiving machine, and from there into the
> receive buffer. I know (or at least have heard) that there is a way to
> cut down on this by "binding" the buffer in the send processes address
> space to one in the receive processes address space and always using
> the same ones, thus you only do one copy instead of three. Mr. Gropp
> mentions this problem sort of in passing on his tutorial web page, but
> isn't real clear about it. Am I safe in assuming that the sort of
> thing you want to do is something involving MPI_Start()s,
> MPI_Send_init()s, MPI_Recv_init()s and MPI_Wait()s?

This isn't something you do explicitly as a user. It is something that
should done automatically by the MPI implementation when possible. 
It will happen with regular MPI_Send/MPI_Recv as well as with
the other modes (except buffered mode). 

On a shared memory machine, a straightforward and simple implementation 
involves only two memory copies - to and from an area of memory
shared by the two processors. In a more sophisticated shared 
memory implementation, you only need one copy - directly from 
the sender's to the receiver's memory. This is done in the new 
implementation from SGI. A send operation updates the receivers
queue but doesn't copy any data. A receive operation copies
the data once. 

On distributed memory machines, you need some direct access
to remote memory to implement the message transfer using a single
copy. I don't know of an implementation that does this. 

One of the advantages of MPI over PVM is that the semantics
of MPI *allow* a single-copy transfer. It is not possible
to do a PVM transfer with fewer than two copies (without 
violating PVM semantics) and PVM requires three copies if
you don't use InPlace packing. 

Bill Saphir 
NAS/NASA Ames


