File locking failed in ADIOI_Set_lock

[Rob Latham]

On Fri, Sep 24, 2010 at 07:49:06AM -0600, Mark Taylor wrote:
> Hi John,
> 
> I've had a very similar issue a while ago on several older Lustre
> filesystems at Sandia, and I can confirm that setting those hints did
> allow the code to run 

if you turn off data sieving then there will be no more lock calls.
Depending on how your application partitions the arrays, that could be
fine, or it could result in a billion 8 byte operations. 

> (but I could never get pnetcdf to be any faster
> than netcdf).  

Unsurprising, honestly.  If you are dealing with Lustre, then you must
both use an updated ROMIO and use collective I/O. 

Here is the current list of MPI-IO implementations that work well with
Lustre:

- Cray MPT 3.2 or newer
- MPICH2-1.3.0a1 or newer
- and that's it. 

I think the OpenMPI community is working on a re-sync with MPICH2
romio.  I also think we can stitch together a patch against OpenMPI if
you really need the improved lustre driver.   I'm not really in
patch-generating mode right now, but maybe once i'm back in the office
I can see how tricky it will be.

> This was with CAM, with pnetcdf being called by PIO, and
> PIO has a compiler option to turn this on, -DPIO_LUSTRE_HINTS.    
> 
> However, on Sandia's redsky (more-or-less identical to RedMesa), I just
> tried these hints and I am also getting those same error messages you
> are seeing. So please let me know if you get this resolved.

I can't think of any other code paths that use locking, unless your
system for some reason presents itself as nfs.  

That's why rajeev suggested prefixing with lustre: Unfortunately, that
won't help: it has only been since March of this year that (with
community support)  the Lustre driver in MPICH2 passed all the ROMIO
tests, and now we need to get that into OpenMPI.

==rob

-- 
Rob Latham
Mathematics and Computer Science Division
Argonne National Lab, IL USA