Newsgroups: comp.parallel.mpi
From: rbarrett@ranger.lanl.gov (Richard Barrett)
Subject: Re: T3D MPI Send/Wait (was Re: PVM send buffers)
Organization: Los Alamos National Laboratory
Date: 1 Apr 1996 15:14:52 GMT
Message-ID: <4jortc$otr@newshost.lanl.gov>

>I must have missed the posting quoted above, some clarification
>is required. CRI/EPCC MPI for Cray T3D is indeed based on CRI's
>SHMEM calls as well as atomic swap functionality, but this does
>not imply that MPI sends are not "safe", if I understand your
>terminology.

I think my terminology is the problem: my definition of a safe send: 
Once my call to send returns control to my program, I am free to write
in to the memory where the data I wanted to send is. This _appears_ to 
not be the case with pvm_send and PvmDataInPlace. (Is it possible that 
although PvmDataRaw/PvmDataDefault works for my code, and PvmDataInPlace
does not, that it is not for the reason I believe, i.e. the data has not
actually been entirely sent?

Please read the rest of my post below...I certainly could use the help.

>Of the MPI send modes available, Synchronous mode does require
>that a send operation complete only after matching a receive 
>posted by the target process. The sender may return while the
>PUT is in progress, but it is guaranteed that all data will be
>written to the target buffer; the receiver ensures data cacheis valid following the transfer.

Since I want asynchronous (pvm_send, MPI_Isend) I'm still reading...

>Standard and Ready send modes (unless forced to synchronise due to
>resource limitations) and Buffered mode copy data into a user/system
>buffer before returning; the receiver fetches the data from this safe
>location using GET, guaranteeing again that the receiver's data
>cache remains valid.

Buffered mode is an option, and is obviously safe, but I'd like to avoid
the data copying...still reading...

>Use of MPI_Wait is not relevant to this discussion. Immediate
>(non-blocking) communications have the same completion requirements
>as blocking only that there is opportunity for user activity between
>posting and completing the operation. For example, an MPI_Wait for
>completion of an MPI_Ibsend *must not* depend on whether a matching
>receive has been posted, or whether the data has actually been
>transferred: it must be a local operation.

Maybe I'm completely off track. I assumed the problem was due to what I described
above. So in using MPI_Isend (asychronous nonbuffered send) I am assuming (uh-oh)
that it behaves similarly. And that's ok, because I have MPI_Wait. But now you're
telling me that's irrelevent! 

I got the idea that MPI_Wait was the solution from page 41 of the MPI Standard
(June 12, 1995 edition): "The functions MPI_Wait and MPI_Test are used to complete
a nonblocking communication. The completion of a send operation indicates that the
sender is now free to update the locations in the send buffer." It goes on 
to mention that this doesn't mean the receive has been satisfied, i.e. synchronicity
is not enforces. 

So I use it as:

   MPI_Isend
   /* do work which doesn't write into the send data */
   MPI_Wait
   /* ok to write into space where sent data is located */

So since I am apparently off base with regards to the t3d, MPI_Wait simply
immediately returns. But since I want portable code, does anyone know of a
machine that requires MPI_Wait with the use of MPI_Isend?

Or maybe I am using the wrong send function:

   MPI_Send: will not return until matching receive posted and data delivered
   MPI_Bsend: asynchronous, but requires data copy. The #1 reason MPI was 
              invented :)
   MPI_Ssend: will not return until matching receive posted
   MPI_Rsend: will only begin when matching receive posted

   MPI_Isend: nonblocking send
   MPI_Ibsend: buffered nonblocking 
   MPI_Issend: synchronous nonblocking
   MPI_Irsend: will send only if a receive is posted?

Anyway, what I've left out is the word "standard" for the description of some
of the above, and that may be the entire source of my confusion. "Standard"
means (p27) that "it is up to MPI to decide whether outgoing messages will
be buffered". So may I assume that the EPCC version of MPI on the T3D buffers
in the standard mode? If so, at what cost? And why not simply use MPI_Wait?

To anyone who read this far, thanks. I sure would appreciated any assistance.
And since MPI is "implementation specific", I would appreciate answers that
address both specific implementations as well as "possible legal implementations".
I really need portable code. Thanks again for the discussions.

Richard

