turning off file locking

[Rajeev Thakur]

The problem with NFS is that by default it does not guarantee that
concurrent writes from two or more processes to different locations
(non-overlapping bytes) in a single file will appear in the file. NFS
clients are allowed to cache data locally at the granularity of a page or
more. To support parallel I/O as defined by MPI-IO, we need to turn off
client-side caching in NFS. There is no reliable way to do that either, but
locking the byte-range being read or written with fcntl file locks for every
read or write seems to work. 

The real recommendation is to not use NFS at all for parallel I/O.

The locking refered to for MPI_MODE_UNIQUE_OPEN under "Advice to
Implementers" is of a different type -- when the MPI-IO implementation
itself does some caching of its own.

Rajeev


> -----Original Message-----
> From: owner-parallel-netcdf at mcs.anl.gov 
> [mailto:owner-parallel-netcdf at mcs.anl.gov] On Behalf Of 
> Drake, Richard R
> Sent: Wednesday, May 19, 2004 5:07 PM
> To: 'parallel-netcdf at mcs.anl.gov'
> Subject: RE: turning off file locking
> 
> 
> I agree that NFS file locking is insufficient, therefore, can 
> it be turned off?  At least when I will only be writing a new file.
> 
> Looking at the MPI-2 spec, the MPI_MODE_UNIQUE_OPEN mode to 
> MPI_File_open() says "...allows an implementation to optimize 
> access by eliminating the overhead of file locking."
> 
> Does pnetcdf open the file more than once using 
> MPI_File_open() and expect MPI-I/O to handle the 
> syncronization correctly, or does pnetcdf open the file 
> uniquely (at least amongh the MPI processes for that job).  
> If so, then why can't file locking be turned off?
> 
> On the other hand, if MPI-I/O will not function correctly 
> without file locking, then there is a bug or lacking feature 
> in the implementation (at least according to MPI-2).
> 
> -rich
> 
> -----Original Message-----
> From: Richard Hedges [mailto:richard-hedges at llnl.gov] 
> Sent: Wednesday, May 19, 2004 3:23 PM
> To: Foucar, James G
> Cc: 'parallel-netcdf at mcs.anl.gov'
> Subject: Re: turning off file locking
> 
> 
> >Hello, I have been having off and on problems with file 
> locking on NFS 
> >when using pnetcdf. The error i get is:
> >
> >File locking failed in ADIOI_Set_lock. If the file system is 
> NFS, you 
> >need to use NFS version 3 and mount the directory with the 'noac' 
> >option (no attribute caching).
> >
> >My question is: is there anyway turn off file locking for 
> the pnetcdf 
> >file?
> >
> >Thanks,
> >James
> 
> 
> Well, I think that the complaint is that the file locking in NFS is 
> not adequate for the parallel operations that you are attempting. 
> Things you can do:
> 
> 1) use a real parallel filesystem (GPFS, CPFS, PVFS, Lustre: not NFS)
> 2) if an SMP node is big enough, use a local filesystem on an 
> SMP node.
> 3) mount NFS without attribute caching (as recommended in the 
> diagnostic message)
> 
> - Richard
> 
> -- 
> 
> 
> 
> ====================================================
> 
> Richard Hedges
> Scalable I/O Project
> Development Environment Group - Livermore Computing
> Lawrence Livermore National Laboratory
> 7000 East Avenue, MS L-550
> Livermore, CA	94551
> 
> v:	(925) 423-2699
> f:	(925) 423-1173
> E:	richard-hedges at llnl.gov
> 
> 
>