Newsgroups: comp.parallel.mpi,comp.parallel.pvm
From: dmb@bifrost.Lanl.GOV (David M. Beazley)
Subject: Re: Thinking about parallel I/O
Organization: T-11, Los Alamos National Laboratory
Date: 11 Jul 1995 15:27:10 GMT
Message-ID: <DMB.95Jul11092710@bifrost.Lanl.GOV>

In article <3tu011$2tur@theory.tc.cornell.edu> jeg@theory.tc.cornell.edu (Jerry Gerner) writes:

>   Parallel I/O is different strokes for different folks, e.g.,
>
>   1) your example of "collecting debug-output" is a good one!  Anyone who's
>      ever tried to debug a parallel program run across N processors using 
>      Brand X's tracing tools probably knows what you're talking about!
>
>   2) John Q. Engineer is modeling unsteady flows around the latest and greatest
>      aircraft surface.  His simulation is running across 128 processors on
>      Brand Y's scalable parallel computer.  Every "N iterations" each and every
>      processor wants to save 18 MB's of data which will be used later on to
>      generate an animation of these flows over time.  "Every N interations"
>      occurs 550 times during a single run!  John Q. has spent some
>      considerable amount of time porting, tuning, tweaking his parallel code
>      in order to get "good speedups".  I bet John Q. wants to have some type
>      of parallel I/O as well in order to deal with 18 MB's x 128 processors x
>      550 occurences = 1.3 TB's of "soon to be viz data"!!!  He probably is 
>      talking about parallel I/O at a fairly fine-grained applications I/O level.
>
>   3) John Q. also spends 1000's of hours of computing time modeling these new
>      aircraft surfaces.  Naturally he's interested in "checkpointing" his runs
>      occasionally (his computer has been known to go down from time to time).
>      When his model checkpoints, each and every processor wants to save 85 MB's
>      of "restart data".  Since John Q. wants to make sure this stuff is around
>      the next time he might wish to restart his simulation, he'd really
>      appreciate having some way to move the 85 MB's x 128 processors = 10.9 GB's
>      of data from Brand Y's filesystem into Brand Z's archival data system.
>      He's probably talking about multiple/parallel (you name 'em,...  HiPPI,
>      whatever) pipes to slam that 10.9 GB's onto (you name 'em,...  RAID, DLT,
>      whatever) some archival media in some mss in some efficient way,...  and to
>      be able to get it back if/when he needs its.
>
>   Just 3 slightly different examples.  The PTOOLS folks might be concerned 
>   about the first of these, the MPI-IO folks about the second, and the PIO folks
>   about the third.  'Course its all just "parallel I/O"   :-)

What a great summary!   I probably should have been more clear in my original message,
but this sums up many of the requirements.   Typically we perform large amounts
of I/O for the purposes of post-processing (visualization or whatever) and checkpointing.
One interesting twist on this seems to be how one handles files generated on different
machines.  It would be really nice if a checkpoint file created on a CM-5 would work
on a T3D, but we can't do this currently due to differences in word size and other
architectural issues.  There's also problems with using files created on different
partition sizes, but perhaps this is an artifact of using message-passing in the
first place (the data-parallel crowd doesn't seem to have a problem with this--at
least not on the CM-5).

Another facet which isn't mentioned above is user I/O through stdin and stdout.
Even though we're running in parallel, we feel that we should be able to have a
certain amount of interaction with the code.  Presently, we use a command intepreter 
that we developed, that allows the user to interactively run the code and perform
diagnostics (debugging, etc...).  It also allows scripting and other features for
batch jobs.   Unfortunately, getting this to work properly across a
variety of machines has been a headache (we ended up writing our own versions of
fscanf, fprintf, and others).   Surely we can't be the only one with a message
passing code that requires interactive I/O at runtime.   How have others handled this? 

To clarify my earlier message, I'm mainly interested in the user-aspects of 
parallel I/O (ie. libraries, etc... ).   After hacking a bunch of I/O code myself
to get our application to run nicely on a few machines, I just keep saying
to myself "someone has got to be working on a better way to do this."    

Again, thanks for everyone's responses.  I'm mainly just looking for interesting
discussion, since this issue has been on my mind for about a year now.

Cheers,

Dave
---------------------------------------
Dave Beazley
Los Alamos National Laboratory
dmb@viking.lanl.gov




     
     










