Newsgroups: comp.parallel.pvm
From: rbarrett@ranger.lanl.gov (Richard Barrett)
Subject: re: send buffers        Steve Barnard writes:
Organization: Los Alamos National Laboratory
Date: 1 Apr 1996 14:41:28 GMT
Message-ID: <4jopuo$oig@newshost.lanl.gov>

>My understanding of shmem (and I've used it a lot) is that a shmem_put
>returns after the data has been injected into the network, and it is then
>OK to reuse the send buffer.  Isn't this safe?  Doesn't this correspond to
>a blocking MPI send?  I have converted shmem codes on the T3D more-or-less
>directly to MPI with no problems.

A blocking send is safe. But I don't want synchronous behavior. And shmem_put
is safe (with the cache flush), but I'm using asynchronous message passing,
which is not simply a shmem_put. The problem is the way PVM is implemented on
the T3D. Not a mistake on the part of CRI, simply a problem for how I want to
use it: PvmDataInPlace is not safe (PvmDataRaw is), and within the PVM library,
I don't think I have a function that will protect me.

>Possibly you are confusing the notion that "shmem_put isn't safe" with the
>correct statement that "shmem_put doesn't enforce cache coherence," but
>there is an easy fix for this.

If I'm using pure shmem, I simply make a t3d function call which ensures
cache coherence. Is there a way to do this with PVM? Even from Cray employees
I've heard more than one explanation for how message passing is implemented
using shmem, so I'm confused. Here are a couple possibilities:

1. The source pe shmem_puts an address and length to the target pe.
   The target pe shmem_gets the data into some buffer space. When the
   target posts its receive, it checks that buffer. If the data is there,
   it copies it where it wants it.

2. The source pe shmem_puts "tells" the target that it wants to give it some
   data. The target shmem_puts to the source pe where it wants the data.
   The source pe shmem_puts it there. The receive works as above.

Neither may be exactly correct. Anyway, for asynchronous message passing
I think #1 makes more sense. Regardless, it's easy to see how shmem is faster
than message passing.

So where do I put the cache flush, and how do I ensure I have flushed what
I really want to (the data)?

With MPI, I'll bet that it was exactly this anticipated situation which led to
MPI_Wait. The amount of synchronicity I get depends on "how far away" I call it
from my next use of the message buffer.

Richard


