<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
Hi Wei-keng,
<div class=""><br class="">
</div>
<div class="">I really should’ve read the documentation more properly (for reference: <a href="https://www.unidata.ucar.edu/software/netcdf/docs/netcdf/NetCDF-64-bit-Offset-Format-Limitations.html#NetCDF-64-bit-Offset-Format-Limitations" class="">https://www.unidata.ucar.edu/software/netcdf/docs/netcdf/NetCDF-64-bit-Offset-Format-Limitations.html#NetCDF-64-bit-Offset-Format-Limitations</a> and <a href="https://trac.mcs.anl.gov/projects/parallel-netcdf/wiki/FileLimits" class="">https://trac.mcs.anl.gov/projects/parallel-netcdf/wiki/FileLimits</a>).
The fact that Pnetcdf 1.5.0 seemed to work without problems really threw me off here. Thanks a lot for the quick clarification!</div>
<div class=""><br class="">
</div>
<div class="">Regards,</div>
<div class=""><br class="">
</div>
<div class="">Michael</div>
<div class=""><br class="">
<div>
<blockquote type="cite" class="">
<div class="">On 01 Dec 2015, at 01:52 , Wei-keng Liao <<a href="mailto:wkliao@eecs.northwestern.edu" class="">wkliao@eecs.northwestern.edu</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">Hi, Michael<br class="">
<br class="">
>From the header, I can see each of 4 variables is of size 1207959552 x 8 bytes = 9 GiB.
<br class="">
Defining variables larger than 4GiB is not allowed in CDF-2 format (i.e. NC_64BIT_OFFSET).<br class="">
There is an exception: only one fixed-size variable can be larger than 4GiB if it is the<br class="">
last variable defined and there is no record variables.<br class="">
<br class="">
PnetCDF 1.5.0 fails to detect this error indicating a bug in PnetCDF.<br class="">
1.6.0 and 1.6.1 should have already fixed this problem.<br class="">
<br class="">
If you would like to define large variables, please consider<br class="">
CDF-5 format, by using NC_64BIT_DATA flag when creating a file.<br class="">
<br class="">
<br class="">
Wei-keng<br class="">
<br class="">
On Nov 30, 2015, at 4:39 PM, Schlottke-Lakemper, Michael wrote:<br class="">
<br class="">
<blockquote type="cite" class="">Hi Wei-keng,<br class="">
<br class="">
The config.log is not easily available for us, since we cannot reproduce the error on our department’s cluster but only on a production system where we do not have direct access to the build system. If it would be helpful, however, I can try to investigate
if we can get hold of it.<br class="">
<br class="">
Setting PNETCDF_SAFE_MODE=1 did not produce any additional output.<br class="">
<br class="">
Below I have attached a header dump of the file that we are trying to write, which was created using Pnetcdf 1.5.0 (which, as reported in my previous mail, works). The file was created - like in the failed case - on two nodes with a total of 48 MPI ranks, and
using "NC_CLOBBER | NC_64BIT_OFFSET” as the file mode. Does this provide you with the information you were looking for? If you need anything else, please let me know!<br class="">
<br class="">
Regards,<br class="">
<br class="">
Michael<br class="">
<br class="">
P.S.: Dump of the header of the file as created with Pnetcdf 1.5.0:<br class="">
<br class="">
<br class="">
netcdf solution_00000000 {<br class="">
dimensions:<br class="">
<span class="Apple-tab-span" style="white-space:pre"></span>dim0 = 1207959552 ;<br class="">
variables:<br class="">
<span class="Apple-tab-span" style="white-space:pre"></span>double variables0(dim0) ;<br class="">
<span class="Apple-tab-span" style="white-space:pre"></span><span class="Apple-tab-span" style="white-space:pre"></span>variables0:name = "u_U" ;<br class="">
<span class="Apple-tab-span" style="white-space:pre"></span>double variables1(dim0) ;<br class="">
<span class="Apple-tab-span" style="white-space:pre"></span><span class="Apple-tab-span" style="white-space:pre"></span>variables1:name = "v_U" ;<br class="">
<span class="Apple-tab-span" style="white-space:pre"></span>double variables2(dim0) ;<br class="">
<span class="Apple-tab-span" style="white-space:pre"></span><span class="Apple-tab-span" style="white-space:pre"></span>variables2:name = "w_U" ;<br class="">
<span class="Apple-tab-span" style="white-space:pre"></span>double variables3(dim0) ;<br class="">
<span class="Apple-tab-span" style="white-space:pre"></span><span class="Apple-tab-span" style="white-space:pre"></span>variables3:name = "p_U" ;<br class="">
<br class="">
// global attributes:<br class="">
<span class="Apple-tab-span" style="white-space:pre"></span><span class="Apple-tab-span" style="white-space:pre"></span>:gridFile = "grid.Netcdf" ;<br class="">
<span class="Apple-tab-span" style="white-space:pre"></span><span class="Apple-tab-span" style="white-space:pre"></span>:blockType = "DG" ;<br class="">
<span class="Apple-tab-span" style="white-space:pre"></span><span class="Apple-tab-span" style="white-space:pre"></span>:timeStep = 0 ;<br class="">
<span class="Apple-tab-span" style="white-space:pre"></span><span class="Apple-tab-span" style="white-space:pre"></span>:time = 0. ;<br class="">
<span class="Apple-tab-span" style="white-space:pre"></span><span class="Apple-tab-span" style="white-space:pre"></span>:meta_creation_user = "xacmicha" ;<br class="">
<span class="Apple-tab-span" style="white-space:pre"></span><span class="Apple-tab-span" style="white-space:pre"></span>:meta_creation_host = "nid07845" ;<br class="">
<span class="Apple-tab-span" style="white-space:pre"></span><span class="Apple-tab-span" style="white-space:pre"></span>:meta_creation_directory = "/lustre/cray/ws7/ws/xacmicha-fabian-0/dg_scaling/testcase-hornet/logs/2015-11-30_20.25.11_01.00" ;<br class="">
<span class="Apple-tab-span" style="white-space:pre"></span><span class="Apple-tab-span" style="white-space:pre"></span>:meta_creation_date = "2015-11-30 20:27:09" ;<br class="">
<span class="Apple-tab-span" style="white-space:pre"></span><span class="Apple-tab-span" style="white-space:pre"></span>:meta_creation_noDomains = 48 ;<br class="">
<span class="Apple-tab-span" style="white-space:pre"></span><span class="Apple-tab-span" style="white-space:pre"></span>:meta_lastModified_user = "xacmicha" ;<br class="">
<span class="Apple-tab-span" style="white-space:pre"></span><span class="Apple-tab-span" style="white-space:pre"></span>:meta_lastModified_host = "nid07845" ;<br class="">
<span class="Apple-tab-span" style="white-space:pre"></span><span class="Apple-tab-span" style="white-space:pre"></span>:meta_lastModified_directory = "/lustre/cray/ws7/ws/xacmicha-fabian-0/dg_scaling/testcase-hornet/logs/2015-11-30_20.25.11_01.00" ;<br class="">
<span class="Apple-tab-span" style="white-space:pre"></span><span class="Apple-tab-span" style="white-space:pre"></span>:meta_lastModified_date = "2015-11-30 20:27:09" ;<br class="">
<span class="Apple-tab-span" style="white-space:pre"></span><span class="Apple-tab-span" style="white-space:pre"></span>:meta_lastModified_noDomains = 48 ;<br class="">
}<br class="">
<br class="">
<blockquote type="cite" class="">On 27 Nov 2015, at 08:19 , Wei-keng Liao <<a href="mailto:wkliao@eecs.northwestern.edu" class="">wkliao@eecs.northwestern.edu</a>> wrote:<br class="">
<br class="">
Hi, Michael<br class="">
<br class="">
Could you please send me your config.log file (from building 1.6.1)?<br class="">
<br class="">
If you describe the variables (number, their dimensions and sizes),<br class="">
it can be helpful. Also, is there any fixed-size variable larger<br class="">
than 2GB?<br class="">
<br class="">
You can set the run-time environment variable PNETCDF_SAFE_MODE to 1<br class="">
to enable the metadata consistency checking in PnetCDF. That might<br class="">
print additional messages in stdout, if an error is detected.<br class="">
<br class="">
Wei-keng<br class="">
<br class="">
On Nov 26, 2015, at 11:04 PM, Schlottke-Lakemper, Michael wrote:<br class="">
<br class="">
<blockquote type="cite" class="">Hi folks,<br class="">
<br class="">
With the 1.6.0/1.6.1 versions of Parallel netCDF, under some conditions we get -62 errors (One or more variable sizes violate format constraints) when working with NC_64BIT_OFFSET files in parallel. It occurs mostly with parallel jobs > 16 MPI ranks (and was
seen with up to 4k ranks so far) and was reproduced both on GPFS as well as Lustre file systems. Other than that, we could not find anything to narrow down the scope of the problem. Our current fix is to use the 1.5.0 version of Parallel netCDF, which has
not yet produced this error, thus from a user perspective this seems like a regression in the 1.6.x series.<br class="">
<br class="">
Any ideas what the problem could be or what we could do to narrow it down?<br class="">
<br class="">
Yours<br class="">
<br class="">
Michael<br class="">
<br class="">
<br class="">
--<br class="">
Michael Schlottke-Lakemper<br class="">
<br class="">
Chair of Fluid Mechanics and Institute of Aerodynamics<br class="">
RWTH Aachen University<br class="">
Wüllnerstraße 5a<br class="">
52062 Aachen<br class="">
Germany<br class="">
<br class="">
Phone: +49 (241) 80 95188<br class="">
Fax: +49 (241) 80 92257<br class="">
Mail: <a href="mailto:m.schlottke-lakemper@aia.rwth-aachen.de" class="">m.schlottke-lakemper@aia.rwth-aachen.de</a><br class="">
Web: <a href="http://www.aia.rwth-aachen.de" class="">http://www.aia.rwth-aachen.de</a><br class="">
<br class="">
</blockquote>
<br class="">
</blockquote>
<br class="">
</blockquote>
<br class="">
</div>
</blockquote>
</div>
<br class="">
</div>
</body>
</html>