          ROMIO: A High-Performance, Portable MPI-IO Implementation

                       Version 1.0.1, July 1998


Major Changes in This Version:
-----------------------------

* This version is included in MPICH 1.1.1 and HP MPI 1.4.

* Added support for NEC SX-4 and created a new device ad_sfs for
  NEC SFS file system.

* New devices ad_hfs for HP/Convex HFS file system and ad_xfs for 
  SGI XFS file system.

* Users no longer need to prefix the filename with the type of 
  file system; ROMIO determines the file-system type on its own.

* Added support for 64-bit file sizes on IBM PIOFS, SGI XFS,
  HP/Convex HFS, and NEC SFS file systems.

* MPI_Offset is an 8-byte integer on machines that support 8-byte integers.
  It is of type "long long" in C and "integer*8" in Fortran.
  With a Fortran 90 compiler, you can use either integer*8 or
  integer(kind=MPI_OFFSET_KIND). 
  If you printf an MPI_Offset in C, remember to use %lld 
  or %ld as required by your compiler. (See what is used in the test 
  program romio/test/misc.c.)

* On some machines, ROMIO detects at configure time that "long long" is 
  either not supported by the C compiler or it doesn't work properly.
  In such cases, configure sets MPI_Offset to long in C and integer in
  Fortran. This happens on Intel Paragon, Sun4, and FreeBSD.

* Added support for passing hints to the implementation via the MPI_Info 
  parameter. ROMIO understands the following hints (keys in MPI_Info object):

  /* on all file systems */ 
     cb_buffer_size - buffer size for collective I/O
     cb_nodes - no. of processes that actually perform I/O in collective I/O
     ind_rd_buffer_size - buffer size for data sieving in independent reads

  /* on all file systems except IBM PIOFS */
     ind_wr_buffer_size - buffer size for data sieving in independent writes
      /* ind_wr_buffer_size is ignored on PIOFS because data sieving 
         cannot be done for writes since PIOFS doesn't support file locking */

  /* on Intel PFS and IBM PIOFS only. These hints are understood only if
     supplied at file-creation time. */
     striping_factor - no. of I/O devices to stripe the file across
     striping_unit - the striping unit in bytes
     start_iodevice - the number of the I/O device from which to start
                      striping (between 0 and (striping_factor-1))

  /* on Intel PFS only. */
     pfs_svr_buf - turn on or off PFS server buffering by setting the value 
               to "true" or "false", case-sensitive.
      
  If ROMIO doesn't understand a hint, or if the value is invalid, the hint
  will be ignored. The values of hints being used by ROMIO at any time 
  can be obtained via MPI_File_get_info.



General Information 
-------------------

ROMIO is a high-performance, portable implementation of MPI-IO (the
I/O chapter in MPI-2). ROMIO's home page is at
http://www.mcs.anl.gov/home/thakur/romio . The MPI-2 standard itself
is available at http://www.mpi-forum.org/docs/docs.html .

This version of ROMIO includes everything defined in the MPI-2 I/O
chapter except shared file pointer functions (Sec. 9.4.4), split
collective data access functions (Sec. 9.4.5), support for file
interoperability (Sec. 9.5), I/O error handling (Sec. 9.7), and I/O
error classes (Sec. 9.8). Since shared file pointer functions are not
supported, the MPI_MODE_SEQUENTIAL amode to MPI_File_open is also not
supported.

The subarray and distributed array datatype constructor functions from
Chapter 4 (Sec. 4.14.4 & 4.14.5) have been implemented. They are
useful for accessing arrays stored in files. The functions
MPI_File_f2c and MPI_File_c2f (Sec. 4.12.4) are also implemented.

C, Fortran, and profiling interfaces are provided for all functions
that have been implemented. 

Please read the limitations of this version of ROMIO that are listed
below (e.g., MPIO_Request object, restriction to homogeneous environments).

This version of ROMIO runs on the following machines: IBM SP; Intel
Paragon; HP Exemplar; SGI Origin2000; NEC SX-4; other symmetric
multiprocessors from HP, SGI, DEC, and IBM; and networks of
workstations (Sun, SGI, HP, IBM, DEC, Linux, and FreeBSD).
Supported file systems are IBM PIOFS, Intel PFS, HP/Convex
HFS, SGI XFS, NEC SFS, NFS, and any Unix file system (UFS).

This version of ROMIO works with and is included in MPICH 1.1.1 and in
HP MPI 1.4. It also works with SGI MPI 3.1 and higher and with NEC MPI.

You can open files on multiple file systems in the same program. The
only restriction is that the directory where the file is to be opened
must be accessible from the process opening the file. For example, a
process running on one workstation may not be able to access a
directory on the local disk of another workstation, and therefore
ROMIO will not be able to open a file in such a directory. NFS-mounted
files can be accessed.

An MPI-IO file created by ROMIO is no different than any other file
created by the underlying file system. Therefore, you may use any of
the commands provided by the file system to access the file, e.g., ls,
mv, cp, rm, ftp.



Installation Instructions
-------------------------

ROMIO is included in MPICH 1.1.1 and gets built as part of MPICH 1.1.1.
ROMIO is also included in HP MPI 1.4 (see http://www.hp.com/go/mpi).
Therefore, if you are using MPICH 1.1.1 or HP MPI 1.4, you need not install
ROMIO separately. If you are using other versions of MPICH or HP MPI,
we encourage you to upgrade to the latest versions, which include ROMIO. 

If you are using SGI MPI or NEC MPI (NEC's version of MPICH), 
you can configure and build ROMIO as follows:

Untar the tar file as

    gunzip -c romio.tar.gz | tar xvf -

OR

    zcat romio.tar.Z | tar xvf -

THEN

    cd romio
    ./configure
    make

Some example programs and a Makefile are provided in the romio/test directory.
Run the examples the way you would run any MPI program. Each program takes 
the filename as a command-line argument "-fname filename".  

The configure script by default configures ROMIO for the file systems
most likely to be used on the given machine. If you wish, you can
explicitly specify the file systems by using the "-file_system" option
to configure. Multiple file systems can be specified by using "+" as a
separator. For example,

    ./configure -file_system=xfs+nfs

For the entire list of options to configure do

    ./configure -h | more

After building a specific version as above, you can install it in a
particular directory with 

    make install PREFIX=/usr/local/romio    (or whatever directory you like)

or just

    make install           (if you used -prefix at configure time)

If you intend to leave ROMIO where you built it, you should NOT install it 
(install is used only to move the necessary parts of a built ROMIO to 
another location). The installed copy will have the include files,
libraries, man pages, and a few other odds and ends, but not the whole
source tree.  It will have a test directory for testing the
installation and a location-independent Makefile built during
installation, which users can copy and modify to compile and link
against the installed copy. 

To rebuild ROMIO with a different set of configure options, do

    make cleanall

to clean everything including the Makefiles created by configure.
Then run configure again with the new options, followed by make.



Testing ROMIO
-------------

To test if the installation works, do

     make testing 

in the romio/test directory. This calls a script that runs the test
programs and compares the results with what they should be. By
default, "make testing" causes the test programs to create files in
the current directory and use whatever file system that corresponds
to. To test with other file systems, you need to specify a filename in
a directory corresponding to that file system as follows:

     make testing TESTARGS="-fname=/foo/piofs/test"



Compiling and Running MPI-IO Programs
-------------------------------------

If you are using SGI MPI or NEC MPI, you need to include the file
mpio.h for C or mpiof.h for Fortran in your MPI-IO program. 
In MPICH 1.1.1 and HP MPI 1.4, these files are already included in
mpi.h and mpif.h, respectively.

Note that on HP machines running HPUX and on NEC SX-4, you need to
compile Fortran programs with mpif90, becuase the f77 compilers on
these machines don't support 8-byte integers. 

With MPICH or HP MPI, you can compile MPI-IO programs as 
   mpicc foo.c
or 
   mpif77 foo.f

With SGI MPI, you can compile MPI-IO programs as 
   cc -Iromio_path/include foo.c romio_path/lib/IRIX64/libmpio.a -lmpi
or
   f77 -Iromio_path/include foo.f romio_path/lib/IRIX64/libmpio.a -lmpi

With NEC MPI, you can compile MPI-IO programs as 
   mpicc -Iromio_path/include foo.c romio_path/lib/SX4/libmpio.a
or
   mpif90 -Iromio_path/include foo.f romio_path/lib/SX4/libmpio.a

Run the program as you would run any MPI program on the machine. If
you use mpirun, make sure you use the correct mpirun for the MPI
implementation you are using. For example, if you are using MPICH on
an SGI machine, make sure that you use MPICH's mpirun and not SGI's
mpirun.

The Makefile in the romio/test directory illustrates how to compile
and link MPI-IO programs. 



Limitations of this version of ROMIO
------------------------------------

* The "status" argument is not filled in any function. Consequently,
MPI_Get_count and MPI_Get_elements will not work when passed the
status object from an MPI-IO operation.

* All nonblocking I/O functions use a ROMIO-defined "MPIO_Request"
object instead of the usual "MPI_Request" object. Accordingly, two
functions, MPIO_Test and MPIO_Wait, are provided to wait and test on
these MPIO_Request objects. They have the same semantics as MPI_Test
and MPI_Wait.

int MPIO_Test(MPIO_Request *request, int *flag, MPI_Status *status);
int MPIO_Wait(MPIO_Request *request, MPI_Status *status);

The usual functions MPI_Test, MPI_Wait, MPI_Testany, etc., will not
work for nonblocking I/O. 

* This version works only on a homogeneous cluster of machines,
and only the "native" file data representation is supported.

* All functions return only two possible error codes---MPI_SUCCESS on
success and MPI_ERR_UNKNOWN on failure.

* On HP machines running HPUX, atomic mode is not supported for files
residing on HFS file systems.

* On HP machines running HPUX and on NEC SX-4, you need to compile
Fortran programs with mpif90 instead of mpif77, because the f77
compilers on these machines don't support 8-byte integers.

* The file-open mode MPI_MODE_EXCL does not work on Intel PFS file system,
due to a bug in PFS.



Usage Tips
----------

* When using ROMIO with SGI MPI, you may sometimes get an error
message from SGI MPI: ``MPI has run out of internal datatype
entries. Please set the environment variable MPI_TYPE_MAX for
additional space.'' If you get this error message, add this line to
your .cshrc file:
    setenv MPI_TYPE_MAX 65536 
Use a larger number if you still get the error message.

* If a Fortran program uses a file handle created using ROMIO's C
interface, or vice-versa, you must use the functions MPI_File_c2f 
or MPI_File_f2c (see MPI-2 Section 4.12.4). Such a situation occurs,
for example, if a Fortran program uses an I/O library written in C
with MPI-IO calls. Similar functions MPIO_Request_f2c and
MPIO_Request_c2f are also provided.

* For Fortran programs on the Intel Paragon, you may need
to provide the complete path to mpif.h in the include statement, e.g.,
        include '/usr/local/mpich/include/mpif.h'
instead of 
        include 'mpif.h'
This is because the -I option to the Paragon Fortran compiler if77
doesn't work correctly. It always looks in the default directories first 
and, therefore, picks up Intel's mpif.h, which is actually the
mpif.h of an older version of MPICH. 



ROMIO Users Mailing List
------------------------

Please register your copy of ROMIO with us by sending email
to majordomo@mcs.anl.gov with the message

subscribe romio-users

This will enable us to notify you of new releases of ROMIO as well as
bug fixes.



Reporting Bugs
--------------

If you have trouble, first check the users guide (in
romio/doc/users-guide.ps.gz). Then check the on-line list of known
bugs and patches at http://www.mcs.anl.gov/home/thakur/romio .
Finally, if you still have problems, send a detailed message containing:

   The type of system (often, uname -a)
   The output of configure
   The output of make
   Any programs or tests

to romio-maint@mcs.anl.gov .



ROMIO Internals
---------------

A key component of ROMIO that enables such a portable MPI-IO
implementation is an internal abstract I/O device layer called
ADIO. Most users of ROMIO will not need to deal with the ADIO layer at
all. However, ADIO is useful to those who want to port ROMIO to some
other file system. The ROMIO source code and the ADIO paper
(doc/adio.ps.gz) will help you get started.
