Newsgroups: comp.sys.transputer
From: raja@osc.edu (Raja Daoud)
Subject: LAM: Trollius on UNIX clusters
Organization: The Ohio Supercomputer Center
Date: 1 Mar 1994 17:41:08 -0500
Message-ID: <2l0ga4$2ak@hawkeye.osc.edu>


Ohio Supercomputer Center (OSC) announces the general release of
LAM, a new open environment for cluster computing.  The software
is available in source code form under the terms of the GNU
general license (V2).  Distribution is via anonymous ftp from
tbag.osc.edu under pub/lam.

The factory default software supports the following platforms:

Sun 4 (sparc), SunOS 4.1.3
Sun 4 (sparc), Solaris 2.3
SGI IRIX 4.0.5
IBM RS/6000, AIX v3r2
DEC AXP, OSF/1 V1.3

Technical Summary

Key Features

- heterogeneous networks
- micro-kernel, multi-process, modular structure
- hands-on monitoring and control
- simple powerful point-to-point communication
- group communication
- easy transition to dedicated h/w
- PVM compatibility
- parallel I/O

Product Rationale

There is no shortage of quality cluster computing environments
available today.  There are commercial as well as free offerings
and PVM from ORNL and partners is an acknowledged de-facto standard.
After boiling away the subtle distinctions (about which we will 
vigorously boast), LAM provides essentially the same capability
as PVM.  Why would anybody expend great effort to duplicate it?
The main reason is that LAM, in an expanded form, already
existed long before the emergence of PVM [Burns 89].  LAM is carved
out of a greater system that combines both clusters and MPPs as one
heterogeneous machine.  We are simply taking an existing product,
removing some features, and releasing it.

Then there are those distinctions.

LAM Internal Structure

LAM has a highly modular micro-kernel, multi-task structure.
Although the environment runs as one UNIX daemon per host, the
daemon is really a ukernel plus a dozen or so "hand-threaded"
system processes.  The ukernel schedules these internal processes
plus any number of external processes (UNIX clients on the same
host) in a prioritized preemptive fashion.  Internal processes
are simply called while external processes, typically application
processes, are replied to through UNIX.  The small ukernel (more
like a nano-kernel) also handles communication between these local
processes.  Internal system processes, not the ukernel, handle
network communication.  Any internal program can be recompiled
and run externally with no change in system behaviour but with an
obvious degradation in performance.  Indeed, all of the internal
processes can be run as separate UNIX processes with the central
daemon consisting of only the ukernel.  We sometimes do system
debugging this way.

LAM enjoys the benefits of modular ukernel design without
the performance disadvantages.  It is trivial to add or subtract
services from LAM without seeing or recompiling the rest
of the system.  This is our justification for calling it
an "open" system.

By replacing a couple of modules full of UNIX system calls
with a couple of modules full of software interrupts, context
switchers and peripheral drivers, LAM can be run natively on
your favourite killer micro as an MPP OS with a real-time
system at the nodes.  OSC does not offer hardware
implementations or support but Alex Informatique
of Montreal does.  Unlike PVM running atop Paragon's native
OS, for example, the native Alex LAM provides the lowest
level of system call portability (ukernel level).  LAM
also has the hooks for handling MPP issues like topology that
are ignored on UNIX networks.

LAM Message Passing

We do not expect too many application programs to use LAM message
passing, since PVM is the de-facto standard.  We support the PVM API.
Point-to-point LAM message passing deals with physical nodes explicitly,
as opposed to PVM global task ID space.  The first thing you put
in a LAM message is the node ID.  Then you worry about which
process on the node will receive it with two other synchronization
variables much like PVM's message type.  Many would see this
explicit nature as a disadvantage but we do not.  Node oriented
communication is quite intuitive for the novice user that understands
workstations and his application, but not too much about process groups,
bulletin boards and other paradigms.  A tuple space user will not care
which nodes contain which tuples, but the system developer will care.
All cluster environments eventually decide explicitly which node gets
a message (node ID is a field in the PVM task ID).  LAM simply
provides a complete API for this layer.

We think LAM's modular structure combined with its low level node
based communication make it the best foundation for building higher
level tools and paradigms.  We've used it to implement PVM
and a tuple space tool, for example.

LAM has a variety of point-to-point messaging choices.

- packetized, routed, traced, buffered
- flow controlled
- virtual circuit, bypassing LAM with a TCP/IP connection
- asynchronous receive
- beginner
- receive probe

LAM also has group communication, much like PVM's.  LAM adds
group receive with reduction functions, group exchange, a eureka
function, and a separate quorum function that blocks the caller
until a certain membership level is reached.

LAM Network Process Management

LAM looks for executables in directories given by the user's PATH variable.
An executable may reside in the target node's file space or LAM can
transfer the file from a third node.

All LAM programs have the usual registration call, along with a polite
de-registration call.  A program can fork[&exec] and re-register as desired.

LAM has its own signal handling and delivering functions that follow the
POSIX model and a distinct set of signals.

LAM I/O

When NFS or friends are not readily available, LAM has a set of
functions that mirror the POSIX functions and provide remote
access to the filesystem of other nodes.  LAM's remote file
server is augmented by two special parallel access methods
[Salmon 87].  They often alleviate the need for single control
programs managing the I/O of a parallel application.

LAM User Interface

LAM's UI is just your favourite shell.  There is no special environment,
only a set of utility programs.

LAM Basic Monitoring

Command and programming interfaces allow the user to closely examine
processes and messages.  Mismatches in synchronization and deadlocks
can easily be spotted.  An extra pair of commands examine processes
and messages within a PVM context.  The status of the parallel remote
filesystem and group membership can also be monitored.

LAM Documentation

There are user guides for C and Fortran programmers as well as
a complete set of manual pages covering every function and command program.

Other LAM Products

If people find LAM useful, other unreleased products associated with
LAM may be made available.  Laying on the shelf somewhere are diagnostic
tools for IPC and I/O, a complete tuple space programming system,
a GUI for basic operations, a graphics daemon, a light show daemon,
and a trace (format indep.) collection and transport system.

-=-

LAM is a subset of Trollius, an environment for UNIX clusters + MPP.
Trollius was originally developed at the Cornell Theory Center.
Trollius is a trademark of The Ohio State University and the Cornell
Research Foundation.

LAM development was sponsored by OSC, OSU, Cray Research Inc., and
Alex Informatique.

-=-

Burns 89	Burns, G., "A Local Area Multicomputer", Proceedings
		of HCCA4, Monterey, 1989

Salmon 87	Salmon, J., "CUBIX: programming hypercubes without
		programming hosts," >>Hypercube Multiprocessors<< 1987,
		edited by M. Heath, SIAM

