<html><head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body>
Hello Lawrence,<br>
<br>
Thanks for the swift response. I am sorry that mine has been much
slower.<br>
<br>
The information that you provided was useful.<br>
<br>
I had been trying to generate the initial state using just one MPI
process. I rewrote my code slightly so that I could use more than
one process. Once I had done this I succeeded in generating an
initial state by employing two MPI processes.<br>
<br>
I do not understand in detail why one process was insufficient
because it seems to me that I was within the stated limits (the
maximum size of file generated using two processes is 2.6GB and the
number of elements is less than 310,000,000) but I do not doubt that
your suspicion is correct.<br>
<br>
Best wishes,<br>
<br>
David<br>
<br>
<div class="moz-cite-prefix">On 26/11/2021 15:25, Lawrence Mitchell
wrote:<br>
</div>
<blockquote type="cite" cite="mid:CA+wRr2kXe2K-pJ_fjESpWktzMWWbFJyzyR67=YkZVBWWRAdN7A@mail.gmail.com">
<div style="background-color:#fff2e6; border:2px dotted #ff884d"><span style="font-size:12pt; font-family: sans-serif; color:black;
font-weight:bold; padding:.2em">This email was sent to you by
someone outside the University.</span>
<div style="font-size:10pt; font-family: sans-serif;
font-style:normal; padding:.2em">
You should only click on links or attachments if you are
certain that the email is genuine and the content is safe.</div>
</div>
<div>
<div dir="auto">This is failing setting the chunksize:
<div><a href="https://gitlab.com/petsc/petsc/-/blob/main/src/dm/impls/da/gr2.c#L517" moz-do-not-send="true" class="moz-txt-link-freetext">https://gitlab.com/petsc/petsc/-/blob/main/src/dm/impls/da/gr2.c#L517</a></div>
<div><br>
</div>
<div dir="auto">It is hard for me to follow this code, but it
looks like the chunk is just set to the total extent of the
DA (on the process?). This can grow too large for HDF5,
which has limits described here <a href="https://support.hdfgroup.org/HDF5/doc/Advanced/Chunking/" moz-do-not-send="true" class="moz-txt-link-freetext">https://support.hdfgroup.org/HDF5/doc/Advanced/Chunking/</a></div>
<div dir="auto"><br>
</div>
<div dir="auto">chunks must have fewer than 2^32-1 elements
and a max size of 4GB</div>
<div dir="auto"><br>
</div>
<div dir="auto">I suspect you’re hitting that limit. </div>
<div dir="auto"><br>
</div>
<div dir="auto">I guess this part of the code should be fully
refactored to set sensible chunk sizes. That same document
suggests 1MB is a good option. </div>
<div dir="auto"><br>
</div>
<div dir="auto">Lawrence</div>
</div>
<div><br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Fri, 26 Nov 2021 at
15:10, David Scott <<a href="mailto:d.scott@epcc.ed.ac.uk" moz-do-not-send="true" class="moz-txt-link-freetext">d.scott@epcc.ed.ac.uk</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left-width:1px;border-left-style:solid;padding-left:1ex;border-left-color:rgb(204,204,204)">Hello,<br>
<br>
I am trying to write out HDF5 files. The program I have
works all right<br>
up to a point but then fails when I double the size of the
file that I<br>
am trying to write out. I have attached a file containing
the error<br>
message and another file containing the short program that
I used to<br>
generate it.<br>
<br>
The first of these files reports that Petsc 3.14.2 was
used. I know that<br>
this is old but I have obtained the same result with Petsc
3.16.1. Also,<br>
I have observed the same behaviour on two different
machines. I tried<br>
using 64 bit indices but it made no difference.<br>
<br>
The program runs successfully with the following command
line options<br>
<br>
-global_dim_x 1344 -global_dim_y 336 -global_dim_z 336<br>
<br>
but fails with these<br>
<br>
-global_dim_x 2688 -global_dim_y 336 -global_dim_z 336<br>
<br>
All the best,<br>
<br>
David<br>
<br>
The University of Edinburgh is a charitable body,
registered in Scotland, with registration number SC005336.
Is e buidheann carthannais a th’ ann an Oilthigh Dhùn
Èideann, clàraichte an Alba, àireamh clàraidh SC005336.<br>
</blockquote>
</div>
</div>
</div>
</blockquote>
<br>
</body>
</html>