[MPICH] tricks for avioding locking problems

[Rob Ross]

Hi Jim,

Do you mind telling us what file system is exhibiting this behavior?  We 
like to know what file systems locks don't work on, so that we can warn 
users.

You can have a look at the User's Guide that is distributed with ROMIO 
for information on this.  We should put a PDF version of that online, 
and also that document is a little out of date.  I'm cc'ing romio-maint 
to remind us of that.

You can use the "romio_ds_read" and "romio_ds_write" hints to control 
data sieving.  Set "romio_ds_write" to "disable" to turn off data 
sieving writes.  "romio_ds_write" does work; the User's Guide is out of 
date.

Note that atomic mode isn't going to work if locks don't work.

As Rajeev mentioned, we have a scheme for eliminating the need for file 
system locks by instead using some MPI-2 features.  However, there are 
some caveats in that approach that preclude us implementing it for the 
moment.  So for now we're stuck with either using file system locks or 
turning off these optimizations (and not using atomic mode).

Regards,

Rob

Foucar, James G wrote:
> OK, I understand what you are saying.
> 
> Which hint disables the data sieving operation?
> Is there a good online resource for hint documentation?
> 
> Thanks for the help!
> -Jim
> 
> -----Original Message-----
> From: Rajeev Thakur [mailto:thakur at mcs.anl.gov]
> Sent: Tue 6/21/2005 3:51 PM
> To: Foucar, James G; mpich-discuss at mcs.anl.gov
> Subject: RE: [MPICH] tricks for avioding locking problems
> 
> Which file system you use matters because different file systems have
> different consistency semantics, and we accordingly have to compensate for
> it in order to provide complete MPI-IO functionality. If you are using NFS,
> you pretty much need locks to work unless you are only doing reads from the
> file. On a true POSIX file system, you need locks to implement shared file
> pointer, the atomic mode, and the data sieving optimization for
> noncontiguous writes (which can be disabled with an info hint). We have come
> up with a way to avoid locks on a POSIX file system by implementing the
> equivalent of locks with MPI one-sided communication, and have 3 papers on
> it, but it is not in the ROMIO release yet. (Locks will still be needed on
> NFS to disable client-side caching.)
> 
> Rajeev
> 
> 
> 
>   _____ 
> 
> From: owner-mpich-discuss at mcs.anl.gov
> [mailto:owner-mpich-discuss at mcs.anl.gov] On Behalf Of Foucar, James G
> Sent: Tuesday, June 21, 2005 4:34 PM
> To: Rajeev Thakur; mpich-discuss at mcs.anl.gov
> Subject: RE: [MPICH] tricks for avioding locking problems
> 
> 
> 
> Let's not focus on NFS at the moment. What I really need is a way, trick, or
> hack that allows me to circumvent or disable file locking in general. Does
> such a thing exist?
> 
> Thanks,
> Jim
> 
> 
> -----Original Message-----
> From: Rajeev Thakur [mailto:thakur at mcs.anl.gov]
> Sent: Tue 6/21/2005 3:25 PM
> To: Foucar, James G; mpich-discuss at mcs.anl.gov
> Subject: RE: [MPICH] tricks for avioding locking problems
> 
> One way to avoid this problem is to not use NFS :-).
> 
> Do you have the lockd daemon running on all nodes?
> 
> Rajeev
> 
> 
>   _____
> 
> From: owner-mpich-discuss at mcs.anl.gov
> [mailto:owner-mpich-discuss at mcs.anl.gov] On Behalf Of Foucar, James G
> Sent: Tuesday, June 21, 2005 4:12 PM
> To: mpich-discuss at mcs.anl.gov
> Subject: [MPICH] tricks for avioding locking problems
> 
> 
> 
> Hello,
> 
> I was wondering if any of you know of ways to avoid the following error:
> 
>  > File locking failed in ADIOI_Set_lock. If the file system is NFS, you
>  > need to use NFS version 3 and mount the directory with the 'noac' option
>  > (no attribute caching).
>  > [0] MPI Abort by user Aborting program !
>  > [0] Aborting program!
>  > p0_928:  p4_error: : 1
>  > exit 1
> 
> This error, and similar errors, have popped up randomly on all many file
> systems, regardless of whether they were mounted with attribute caching.
> 
> I have tried using the MPI_MODE_UNIQUE_OPEN flag when opening files and some
> other tricks, but nothing seems to get rid of this problem permanently on
> all the various platforms that I need to support.
> 
> What I really need is a general way to tell MPICH not to use any file
> locking or at least some way to avoid this error.
> 
> Thanks!
> -Jim
> 
> 
> 
> 
>