Newsgroups: comp.parallel.pvm
From: fsset@lerc.nasa.gov (Scott Townsend)
Subject: Re: solution to bad performance of pvm on SP2
Organization: NASA Lewis Research Center
Date: 11 Jan 1996 10:59:59 -0500
Message-ID: <4d3c5v$1bj3@lace.lerc.nasa.gov>

In article <4d0fh3$i0t@mira.sara.nl>, Willem Vermin  <sond0074> wrote:
>We found that pvm has a very bad performance transmitting blocks larger
>than 16Kbyte on an SP2, PVM_ARCH=RS6K, running AIX 4.1.4. (pvm level 3.3.10)
>
>Solution:
>
>in src/lpvm.c:
>
>replace
>
>#define TTSOCKBUF       0x8000
>
>by
>
>#define TTSOCKBUF       0x40000
>
>Willem Vermin
>SARA
>willem@sara.nl
>

I expanded on this a bit some months ago while doing some performance tests
of an ATM link over a satellite.  This gets called before connecting or
accepting a connection.  The only significant differences between the
(then 3.3.7) lpvm.c code is the PVM_SOCKBUF environment variable to avoid
a recompile and using WINSHIFT on those systems which don't automagically
handle buffer sizes over 64KB.  (it's been put into a subroutine to avoid
some maintenance headaches)

This was tested with PVM 3.3.7 between an SGI5 and a CRAY YMP.  I would expect
this same scheme to work with later versions & other UNIX-like machines.
This of course only works for RouteDirect communications.

/*
 *  Set various options on a TCP (AF_INET) socket.
 *  Large SO_SNDBUF/SO_RCVBUF is especially important on satellite links.
 *  (s.townsend@lerc.nasa.gov)
 */
pvmQuickTCP(s, str)
	int	s;
	char	*str;
{
#ifndef NOSOCKOPT

	int	one, bufsiz, winshift;
	char	*bufstr, errbuf[80];

	/* TCP_NODELAY - don't wait to fill a TCP message before sending */

	one = 1;
	if (setsockopt(s, IPPROTO_TCP, TCP_NODELAY,
			(char *)&one, sizeof(one)) == -1) {
		strcpy(errbuf, str);
		strcat(errbuf, " setsockopt TCP_NODELAY");
		pvmlogperror(errbuf);
	}

	/* grab buffer size environment variable if it exists */

	if (((bufstr = getenv("PVM_SOCKBUF")) == NULL) ||
	    ((bufsiz = atoi(bufstr)) <= 0))
		bufsiz = TTSOCKBUF;

#ifdef TCP_WINSHIFT

	/* TCP_WINSHIFT - needed on systems which don't automagically handle
			  buffer sizes larger than 16 bits (i.e. CRAY) */

	winshift = 0;
	while (bufsiz > 0XFFFF) {
		bufsiz >>= 1;
		++winshift;
	}

	if (winshift > 0) {
		if (setsockopt(s, IPPROTO_TCP, TCP_WINSHIFT,
				(char *)&winshift, sizeof(winshift)) == -1) {
			strcpy(errbuf, str);
			strcat(errbuf, " setsockopt TCP_WINSHIFT");
			pvmlogperror(errbuf);
		}
	}

#endif

	/* socket send & receive buffer sizes */

	if (setsockopt(s, SOL_SOCKET, SO_SNDBUF,
			(char *)&bufsiz, sizeof(bufsiz)) == -1) {
		strcpy(errbuf, str);
		strcat(errbuf, " setsockopt SO_SNDBUF");
		pvmlogperror(errbuf);
	}

	if (setsockopt(s, SOL_SOCKET, SO_RCVBUF,
			(char *)&bufsiz, sizeof(bufsiz)) == -1) {
		strcpy(errbuf, str);
		strcat(errbuf, " setsockopt SO_RCVBUF");
		pvmlogperror(errbuf);
	}

#endif
}


-- 

Scott Townsend, NYMA Inc. at the NASA Lewis Research Center 
s.townsend@lerc.nasa.gov

