Newsgroups: comp.parallel.mpi
From: Stephen Barnard <steve@megafauna.com>
Reply-To: steve@megafauna.com
Subject: Re: T3D MPI Send/Wait (was Re: PVM send buffers)
Organization: megafauna
Date: Sat, 06 Apr 1996 21:11:34 -0800
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Message-ID: <31674E86.504A@megafauna.com>

Richard Barrett wrote:
> 
> >I must have missed the posting quoted above, some clarification
> >is required. CRI/EPCC MPI for Cray T3D is indeed based on CRI's
> >SHMEM calls as well as atomic swap functionality, but this does
> >not imply that MPI sends are not "safe", if I understand your
> >terminology.
> 
> I think my terminology is the problem: my definition of a safe send:
> Once my call to send returns control to my program, I am free to write
> in to the memory where the data I wanted to send is. This _appears_ to
> not be the case with pvm_send and PvmDataInPlace. (Is it possible that
> although PvmDataRaw/PvmDataDefault works for my code, and PvmDataInPlace
> does not, that it is not for the reason I believe, i.e. the data has not
> actually been entirely sent?
> 
> Please read the rest of my post below...I certainly could use the help.
> 
> >Of the MPI send modes available, Synchronous mode does require
> >that a send operation complete only after matching a receive
> >posted by the target process. The sender may return while the
> >PUT is in progress, but it is guaranteed that all data will be
> >written to the target buffer; the receiver ensures data cacheis valid following the transfer.
> 
> Since I want asynchronous (pvm_send, MPI_Isend) I'm still reading...
> 
> >Standard and Ready send modes (unless forced to synchronise due to
> >resource limitations) and Buffered mode copy data into a user/system
> >buffer before returning; the receiver fetches the data from this safe
> >location using GET, guaranteeing again that the receiver's data
> >cache remains valid.
> 
> Buffered mode is an option, and is obviously safe, but I'd like to avoid
> the data copying...still reading...
> 
> >Use of MPI_Wait is not relevant to this discussion. Immediate
> >(non-blocking) communications have the same completion requirements
> >as blocking only that there is opportunity for user activity between
> >posting and completing the operation. For example, an MPI_Wait for
> >completion of an MPI_Ibsend *must not* depend on whether a matching
> >receive has been posted, or whether the data has actually been
> >transferred: it must be a local operation.
> 
> Maybe I'm completely off track. I assumed the problem was due to what I described
> above. So in using MPI_Isend (asychronous nonbuffered send) I am assuming (uh-oh)
> that it behaves similarly. And that's ok, because I have MPI_Wait. But now you're
> telling me that's irrelevent!
> 
> I got the idea that MPI_Wait was the solution from page 41 of the MPI Standard
> (June 12, 1995 edition): "The functions MPI_Wait and MPI_Test are used to complete
> a nonblocking communication. The completion of a send operation indicates that the
> sender is now free to update the locations in the send buffer." It goes on
> to mention that this doesn't mean the receive has been satisfied, i.e. synchronicity
> is not enforces.
> 
> So I use it as:
> 
>    MPI_Isend
>    /* do work which doesn't write into the send data */
>    MPI_Wait
>    /* ok to write into space where sent data is located */
> 
> So since I am apparently off base with regards to the t3d, MPI_Wait simply
> immediately returns. But since I want portable code, does anyone know of a
> machine that requires MPI_Wait with the use of MPI_Isend?
> 
> Or maybe I am using the wrong send function:
> 
>    MPI_Send: will not return until matching receive posted and data delivered
>    MPI_Bsend: asynchronous, but requires data copy. The #1 reason MPI was
>               invented :)
>    MPI_Ssend: will not return until matching receive posted
>    MPI_Rsend: will only begin when matching receive posted
> 
>    MPI_Isend: nonblocking send
>    MPI_Ibsend: buffered nonblocking
>    MPI_Issend: synchronous nonblocking
>    MPI_Irsend: will send only if a receive is posted?
> 
> Anyway, what I've left out is the word "standard" for the description of some
> of the above, and that may be the entire source of my confusion. "Standard"
> means (p27) that "it is up to MPI to decide whether outgoing messages will
> be buffered". So may I assume that the EPCC version of MPI on the T3D buffers
> in the standard mode? If so, at what cost? And why not simply use MPI_Wait?
> 
> To anyone who read this far, thanks. I sure would appreciated any assistance.
> And since MPI is "implementation specific", I would appreciate answers that
> address both specific implementations as well as "possible legal implementations".
> I really need portable code. Thanks again for the discussions.
> 
> Richard

My two cents worth is that you should expect to be able to rely on the 
MPI and PVM specifications, and if the specifications are ambiguous then 
they aren't worth a damn.

Personally, I've had no trouble implementing PVM and MPI codes on the 
T3D, but maybe I haven't faced the same issues you are dealing with.

	Steve Barnard

