<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div class="">It is not recommended to use NFS when performing parallel I/O.</div>
<div class="">Different kinds of errors may pop up, due to NFS's client-side</div>
<div class="">aggressive caching. Performance can also be poor if using NFS.</div>
<br class="">
<div class="">Wei-keng </div>
<div><br class="">
<blockquote type="cite" class="">
<div class="">On Jul 12, 2022, at 3:41 AM, Lukas Umek <<a href="mailto:lukas.umek@gmail.com" class="">lukas.umek@gmail.com</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div dir="ltr" class="">Hello Wei-King,
<div class="">Thanks for the pointing this out! At the moment we are using an NFS where WRF is running but we are planning to try out Lustre soon.</div>
<div class="">I did set export ROMIO_TUNEGATHER=0 but this had no obvious effect. </div>
<div class=""><br class="">
</div>
<div class="">best,</div>
<div class="">Luaks </div>
</div>
<br class="">
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">Am Do., 7. Juli 2022 um 17:37 Uhr schrieb Wei-Keng Liao <<a href="mailto:wkliao@northwestern.edu" class="">wkliao@northwestern.edu</a>>:<br class="">
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">
Hi, Lukas<br class="">
<br class="">
The error message points to MPI_Allgather, but PnetCDF does not call MPI_Allgather<br class="">
internally. This most likely came from the MPI-IO library. What file system<br class="">
are you using when running WRF? That can narrow down to the source codes in<br class="">
MPI-IO. In the meantime, can you try setting the following environment variable?<br class="">
<br class="">
export ROMIO_TUNEGATHER=0<br class="">
<br class="">
Wei-keng<br class="">
<br class="">
> On Jul 7, 2022, at 7:17 AM, Lukas Umek <<a href="mailto:lukas.umek@gmail.com" target="_blank" class="">lukas.umek@gmail.com</a>> wrote:<br class="">
> <br class="">
> Hi, <br class="">
> I am using PnetCDF v1.12.2 read&write large netCDF files (64bit offset and CDF5 formats, > 10GB per file) with the WRF model. This works fine up to a certain number of MPI processes. Running on 4080 MPI processes works but a job with 4200 MPI processes fails
during I/O. An example for the error message I get is below: <br class="">
> <br class="">
> Invalid error code (-1) (error ring index 127 invalid)<br class="">
> INTERNAL ERROR: invalid error code ffffffff (Ring Index out of range) in MPIDI_NM_mpi_allgather:202<br class="">
> Abort(873534479) on node 1450 (rank 1450 in comm 0): Fatal error in PMPI_Allgather: Other MPI error, error stack:<br class="">
> PMPI_Allgather(401)..........................: MPI_Allgather(sbuf=0x7ffc94b87a48, scount=1, MPI_LONG_LONG_INT, rbuf=0xd1bba70, rcount=1, datatype=MPI_LONG_LONG_INT, comm=comm=0xc400001a) failed<br class="">
> MPIDI_Allgather_intra_composition_alpha(1844):<br class="">
> MPIDI_NM_mpi_allgather(202)..................:<br class="">
> <br class="">
> This is happening with Intel MPI 2019.9 and 2021.2. When I use mvapich2-2.3.5 <br class="">
> I am able to write files with PnetCDF with more MPI processes involved (e.g. I tried up to 5760 MPI processes and that worked). However performance is much degraded when using mvapich so this is not really an option (time for writing to the disks more than
triples compared to jobs using intelMPI with the same core count and data). <br class="">
> <br class="">
> My problem sounds similar to some threads I found online:<br class="">
> - <a href="https://urldefense.com/v3/__https://lists.mcs.anl.gov/pipermail/parallel-netcdf/2013-August/001519.html__;!!Dq0X2DkFhyF93HkjWTBQKhk!TQVGtYvdEZg4k-8B7Zi6sWhaivDzMcl2cBdRTlt-0Z-f5CdB6KhX4wHA8wR8NJKq-KZYgbSj6QtroHEfFoMxAydI$" rel="noreferrer" target="_blank" class="">
https://lists.mcs.anl.gov/pipermail/parallel-netcdf/2013-August/001519.html</a> <br class="">
> - <a href="https://urldefense.com/v3/__https://lists.mcs.anl.gov/pipermail/parallel-netcdf/2010-October/001143.html__;!!Dq0X2DkFhyF93HkjWTBQKhk!TQVGtYvdEZg4k-8B7Zi6sWhaivDzMcl2cBdRTlt-0Z-f5CdB6KhX4wHA8wR8NJKq-KZYgbSj6QtroHEfFpn68yjZ$" rel="noreferrer" target="_blank" class="">
https://lists.mcs.anl.gov/pipermail/parallel-netcdf/2010-October/001143.html</a><br class="">
> (Setting the MPI_TYPE_MAX suggested in the second post did not help with my problem.)
<br class="">
> <br class="">
> Is anybody aware of some limitations intelMPI imposes when used with PnetCDF? <br class="">
> <br class="">
> cheers,<br class="">
> Lukas <br class="">
<br class="">
</blockquote>
</div>
</div>
</blockquote>
</div>
<br class="">
</body>
</html>