<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body bgcolor="#ffffff" text="#000000">
Hi Wei-keng,<br>
<br>
Thanks for the response. By "independent" do you mean
non-collective? If so, then yes, this version of the code is using
the forms of the API that do not have "_ALL" at the end of the
routine names. The Sandia RedMesa system is not open,
unfortunately. It is Infiniband and running OpenMPI. I received
the following response to Rajeev's suggestion from the RedMesa
admins yesterday:<br>
<blockquote>
<pre class="literal-block">Running a global “flock” mount option has the potential to
introduce file system instability and If this is a mandatory
requirement for this code to run, I would recommend that we do
this as part of the tail end of a system time. If things look ok,
then we can enable this as part of your production environment
(its a fairly easy change). None of our core MPI codes use this
feature, so we have little experience with it, but we are always
willing to learn.
</pre>
</blockquote>
So they're willing to try this, but it'll have to wait til the next
"system time" (I assume that means the next maintenance period).
Does anyone in the pNetCDF group have experience using this option
on Lustre? Could it impact stability?<br>
<br>
I will try the suggested MPI-IO hints. Thanks very much. <br>
<br>
John<br>
<br>
On 9/23/2010 7:31 PM, Wei-keng Liao wrote:
<blockquote
cite="mid:B0FBF4AA-E7E9-4D8F-AC7D-D39BD0E674DD@ece.northwestern.edu"
type="cite">
<pre wrap="">Hi, John,
In addition to the suggestion from Rajeev, could you try use the following MPI-IO hints?
MPI_Info_set(info, "romio_ds_write", "disable");
MPI_Info_set(info, "romio_ds_read", "disable");
One question, does your WRF call pnetcdf independent APIs? I suspect this error
occurs in an independent API. Can you verify that?
I could not access redmesa.sandia.gov (is it open to public?).
Is it running IB and hence mvapich?
Wei-keng
On Sep 23, 2010, at 3:43 PM, Rajeev Thakur wrote:
</pre>
<blockquote type="cite">
<pre wrap="">Also try prefixing the file name with lustre: if you can and see if it makes any difference.
Rajeev
On Sep 23, 2010, at 3:40 PM, Rajeev Thakur wrote:
</pre>
<blockquote type="cite">
<pre wrap="">Based on a user's feedback, we have recently updated that error message to say
"If the file system is LUSTRE, ensure that the directory is mounted with the 'flock' option."
Rajeev
On Sep 23, 2010, at 3:23 PM, John Michalakes wrote:
</pre>
<blockquote type="cite">
<pre wrap="">Hi List,
I'm running WRF on a large cluster with a Lustre file system (redmesa.sandia.gov) and am encountering the following error on one of the processes doing a write through pNetCDF over OpenMPI:
File locking failed in ADIOI_Set_lock(fd 18,cmd F_SETLKW/7,type F_WRLCK/1,whence 0) with return value FFFFFFFF and errno 26.
If the file system is NFS, you need to use NFS version 3, ensure that the lockd daemon is running on all the machines, and mount the directory with the 'noac' option (no attribute caching).
ADIOI_Set_lock:: Function not implemented
ADIOI_Set_lock:offset 74588, length 804
Have Googled this a little before writing to this list, and the workaround seems to be to upgrade to NFS v3, but that's outside my purview and I don't know if that's applicable to this Lustre system anyway. Have you seen this before? Any thoughts or suggestions? Thanks,
John
--
John Michalakes
National Renewable Energy Laboratory
1617 Cole Blvd.
Golden, Colorado 80401
Phone: 303-275-4297
Fax: 303-275-4091
<a class="moz-txt-link-abbreviated" href="mailto:John.Michalakes@nrel.gov">John.Michalakes@nrel.gov</a>
</pre>
</blockquote>
<pre wrap="">
</pre>
</blockquote>
<pre wrap="">
</pre>
</blockquote>
<pre wrap="">
</pre>
</blockquote>
<br>
<pre class="moz-signature" cols="72">--
John Michalakes
National Renewable Energy Laboratory
1617 Cole Blvd.
Golden, Colorado 80401
Phone: 303-275-4297
Fax: 303-275-4091
<a class="moz-txt-link-abbreviated" href="mailto:John.Michalakes@nrel.gov">John.Michalakes@nrel.gov</a>
</pre>
</body>
</html>