PnetCDF, ROMIO & "noac" setting

[Latham, Robert J.]

On Tue, 2018-04-17 at 08:04 -0500, Carl Ponder wrote:
> On 04/16/2018 11:25 PM, Wei-keng Liao wrote:
> > > Using NFS is discouraged for MPI-IO (and PnetCDF as well)
> > > applications, because
> > > it requires a special setting in NFS lock daemons to make it work
> > > correctly for MPI-IO.
> > > See ROMIO user guide at ftp://ftp.mcs.anl.gov/pub/romio/users-gui
> > > de/node12.html
> >  
> > > If the HPC cluster you are using has a parallel file system, such
> > > as Lustre or
> > > OrangeFS, then you should use it.
> 
> Wei-keng -- my sysadmin's tell me that NFS is all that we're ever
> going to have on our cluster.
> I can live with substandard I/O performance, I just want the various
> I/O libraries to give the right data.

I would do the following:

NFS is not a parallel file system.  Read-only from NFS might be
OK.  Writing is trickier because NFS implements bizzare caching
rules.  Sometimes the clients hold onto data for a long time, even
after you ask every way possible to plase write this data out to disk.

I would use collective I/O, but then tell the MPI-IO library to carry
out all I/O from just one rank.   For ROMIO if you set the hint
"cb_nodes" to "1", then all clients will use a single aggregator to
write, and you can hopefully sidestep any caching problems.

You should also set "romio_cb_wrtie" to "enable" to force all writes,
even ones ROMIO might think are nice friendly large contiguous
requests, to take the two-phase collective buffering path.


> We can try following these ROMIO instructions, but can you answer
> some questions about it?
> 1) It says that ROMIO is used in MPICH. Just to confirm, do MVAPICH2
> & OpenMPI & Intel/MPI all use ROMIO?
>     And even if not, would the instructions still be relevant to
> these other MPI's?

Yes.  OpenMPI now has OMPIO, their own MPI-IO implementation.  I know
that you can tune OMPIO in similar ways to ROMIO, but I don't know the
specifics.  I also know you can still request ROMIO.

> 2) It talks about NFS version 3. It looks like we have a combination
> of 3 & 4 on our system.
>      Are the instructions relevant to NFS version 4?

yes

> 3) Given the reservations about performance, I suppose I could ask
> for a special directory to be created & mounted for "coherent"
> operations, and leave the other mounts as-is.
>     Do you see any problems with doing this?
> Thanks,
>                    Carl

no problems with that approachs.  Even with as much caching turned of
as possbile, NFS is still an awkward fit for parallel workloads,
though.  I'm pretty seriously considering removing it from ROMIO.

==rob