Newsgroups: comp.parallel.pvm
From: rbarrett@ranger.lanl.gov (Richard Barrett)
Subject: PVM buffering
Organization: Los Alamos National Laboratory
Date: 4 Mar 1996 18:18:30 GMT
Message-ID: <4hfc5m$9et@newshost.lanl.gov>

Folks,

I brought this up at the recent PVM meeting, but I didn't do a good job
of explaining what I really wanted, so here's another try:

pvm_pack functions are convenient. Convenience in programming usually 
leads to lower performance, and this is especially true in this case.
I am programming a 3d hydrodynamics code using a cell based approach, where
cells are represented by C structs. This is quite convenient in terms of
physics programming. It is quite inconvenient in terms of passing interior
domain boundaries in that the data is scattered about memory. Hundreds of
calls to pvm_pack per time step leads to an obvious lack of performance. 
Easy solution: do my own memory management, resulting in a contiguous buffer of
data. Obviously use PvmDataInPlace buffering, right? Except that with the
Cray T3D, the send is not safe, i.e. the call to pvm_send returns before
the data has been received by the target processor. (Message passing on the
T3D uses shmem. The source pe shmem_puts _information_ to the target pe that 
tells the target where to find the data. The target then does a shmem_get to 
retrieve the data. The sender returns after the shmem_put. At least this is how a
CRI employee explained it to me. Regardless PvmDataInPlace fails for my code
whereas PvmDataRaw works.) Therefore I must use PvmDataRaw, resulting
in a data copy: wasted time and just as important to us, wasted memory. 

A solution would be to have a function that would let me attach my buffer
pointer to the send buffer used by PVM, i.e. they would be the same thing.
It would be convenient if this function would also malloc the space for me.
A bonus would be that I could examine the actual contents of the buffer using 
a debugger without resorting to using the PVM debug stuff. (MPI-ers will 
recognize this funciton as MPI_BUFFER_ATTACH.) 

Anyway, I talked to Bob Manchek about this, and he seemed interested. However,
I would like to see if others run into these problems. 

1. Would others like the buffer control?

2. Does anyone know if other architectures use unsafe sends with PvmDataInPlace?
   What about SMPs? If only the T3D has this situation, perhaps we could convince
   CRI to at least provide a pvm_data_safe function (or whatever). (We don't
   necessarily want a pvm_safe_send as that would force synchronicity, or
   something equally as unwanted.)

BTW, when we switched from pvm_packing to our own memory management, we got a
25 times speed up on the T3D. 

Thanks for any input.

Richard

