Newsgroups: comp.parallel.mpi
From: jamie@lumen.acl.lanl.gov (James Painter)
Subject: Re: T3D MPI Send/Wait (was Re: PVM send buffers)
Organization: Los Alamos National Laboratory
Date: 04 Apr 1996 01:53:57 GMT
Message-ID: <JAMIE.96Apr3185357@lumen.acl.lanl.gov>



> I used the quoted method above, but it does not work as expected - The
> target PE which calls shmem_wait returns from that call before it receives
> the data from the source PE which calls shmem_put :(  Therefor I could not
> do a simple conversion from the Paragon csend/crecv to a T3D
> shmem_put/shmem_wait.  Any insight on "my" problem would be greatly
> appreciated.  Thank you.

SHMEM is not a message passing API so a one-for-one replacement for
Intel csend/crecv is going to take a fair bit of work to deal with
synchronization and buffer management issues.

A couple of things you should be aware of if you want to use SHMEM
directly:

First, there are some compiler directives to make sure the write buffers
are flushed and to wait until the transmitted data gets across the net
and is acknowledged:

  /* Wait for the write to complete */
  _memory_barrier();  
  _remote_write_barrier();

Alternatively, use the shmem_quiet() function.

These need to be called by the sender after the shmem_put if you
want the sender to wait until it knows the data has arrived
at the receiver before continuing.

If you want the receiver to block until the data arrives, you
need to have the sender set a flag word and have the receiver
spin on that flag word until the value changes.

For example, the sender does:

  flag = DONE;
  shmem_put(&flag, &flag, 1, destination );

after the data is certain to have arrived at the receiver.

and the receiver does something like:

  do
    shmem_udcflush_line( (void *) &flag));
  while (flag != DONE);

waiting for the data to arrive.

Of course, if a receiver may be receiving from multiple senders at the
same time you'd have to make flag an array (indexed by sender) instead
of a single scalar.  Also, you'll have to do your own buffer
management for receive buffers.

Finally, if the receiver doesn't have the cache invalidate filter on,
it will have to explicitly flush the cache after the receive to make
sure it isn't seeing stale data in the cache.

--Jamie


