Newsgroups: comp.parallel.mpi
From: A Gordon Smith <smith>
Subject: T3D MPI Send/Wait (was Re: PVM send buffers)
Organization: Department of Computer Science, Edinburgh University
Date: Fri, 29 Mar 1996 17:21:20 GMT
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Message-ID: <Dp1HJL.Ao3.0.staffin.dcs.ed.ac.uk@dcs.ed.ac.uk>

mfergus@relay.nswc.navy.mil (Michael Ferguson) wrote:
>In Article <4j6cva$qtk@newshost.lanl.gov>, rbarrett@ranger.lanl.gov (Richard
>Barrett) wrote:
>>
>>Our problem is that even though our code is portable, our target machine these
>>days is a Cray T3D. Message passing is built on top of its shmem stuff, so
>>sends are not safe (i.e. the send returns before all the data has been sent
>>to the target process). So we're experimenting with MPI_Wait (blocks until the
>>data has been sent vs. the buffering scheme. And as usual, this will be 
>>MPI implementation dependent as well as problem dependent.
>>

I must have missed the posting quoted above, some clarification
is required. CRI/EPCC MPI for Cray T3D is indeed based on CRI's
SHMEM calls as well as atomic swap functionality, but this does
not imply that MPI sends are not "safe", if I understand your
terminology.

Of the MPI send modes available, Synchronous mode does require
that a send operation complete only after matching a receive 
posted by the target process. The sender may return while the
PUT is in progress, but it is guaranteed that all data will be
written to the target buffer; the receiver ensures data cache
is valid following the transfer.

Standard and Ready send modes (unless forced to synchronise due to
resource limitations) and Buffered mode copy data into a user/system
buffer before returning; the receiver fetches the data from this safe
location using GET, guaranteeing again that the receiver's data
cache remains valid.

Use of MPI_Wait is not relevant to this discussion. Immediate
(non-blocking) communications have the same completion requirements
as blocking only that there is opportunity for user activity between
posting and completing the operation. For example, an MPI_Wait for
completion of an MPI_Ibsend *must not* depend on whether a matching
receive has been posted, or whether the data has actually been
transferred: it must be a local operation.


>
>The above last remark by Barrett reminded me of a related problem I ran into
>when trying to port a program written for the Intel Paragon to the Cray T3D.
>I did not use MPI, instead, I tried using the "shmem stuff".  I also ran
>into a similar problem with the 'shmem_wait' which I presume 'MPI_Wait'
>uses, so I am very interested in the results of the 'MPI_Wait' experiment -
>did it work?


In fact, MPI_Wait does not use shmem_wait. It must instead progress all
other outstanding communications while it checks for completion of the
given operation.

The one-sided communications model provided by SHMEM is very different
to the traditional message-passing provided by MPI. I assume that Richard
was talking about MPI_Wait of an Immediate send, called by the source
process. With SHMEM, I assume you have shmem_wait called by the target 
process to wait for a shmem_put transfer. One-sided communications is
part of the MPI 2 proposed standard for extension of MPI; the current
proposal has waiting and testing for an MPI_Put transfer provided
by using MPI_Wait* or MPI_Test* on a special counter MPI_Request
incremented at the target only after completion of the remote write.


-- 
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
 -=-=- A. Gordon Smith -=- Edinburgh Parallel Computing Centre -=-=-
 =- http://www.epcc.ed.ac.uk/~smith -=- Phone +44 (0)131 650 6712 -=
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=


