[petsc-users] Error Writing Large HDF5 File

[David Scott]

Hello Lawrence,

Thanks for the swift response. I am sorry that mine has been much slower.

The information that you provided was useful.

I had been trying to generate the initial state using just one MPI 
process. I rewrote my code slightly so that I could use more than one 
process. Once I had done this I succeeded in generating an initial state 
by employing two MPI processes.

I do not understand in detail why one process was insufficient because 
it seems to me that I was within the stated limits (the maximum size of 
file generated using two processes is 2.6GB and the number of elements 
is less than 310,000,000) but I do not doubt that your suspicion is correct.

Best wishes,

David

On 26/11/2021 15:25, Lawrence Mitchell wrote:
> This email was sent to you by someone outside the University.
> You should only click on links or attachments if you are certain that 
> the email is genuine and the content is safe.
> This is failing setting the chunksize:
> https://gitlab.com/petsc/petsc/-/blob/main/src/dm/impls/da/gr2.c#L517
>
> It is hard for me to follow this code, but it looks like the chunk is 
> just set to the total extent of the DA (on the process?). This can 
> grow too large for HDF5, which has limits described here 
> https://support.hdfgroup.org/HDF5/doc/Advanced/Chunking/
>
> chunks must have fewer than 2^32-1 elements and a max size of 4GB
>
> I suspect you’re hitting that limit.
>
> I guess this part of the code should be fully refactored to set 
> sensible chunk sizes. That same document suggests 1MB is a good option.
>
> Lawrence
>
> On Fri, 26 Nov 2021 at 15:10, David Scott <d.scott at epcc.ed.ac.uk> wrote:
>
>     Hello,
>
>     I am trying to write out HDF5 files. The program I have works all
>     right
>     up to a point but then fails when I double the size of the file that I
>     am trying to write out. I have attached a file containing the error
>     message and another file containing the short program that I used to
>     generate it.
>
>     The first of these files reports that Petsc 3.14.2 was used. I
>     know that
>     this is old but I have obtained the same result with Petsc 3.16.1.
>     Also,
>     I have observed the same behaviour on two different machines. I tried
>     using 64 bit indices but it made no difference.
>
>     The program runs successfully with the following command line options
>
>     -global_dim_x 1344 -global_dim_y 336 -global_dim_z 336
>
>     but fails with these
>
>     -global_dim_x 2688 -global_dim_y 336 -global_dim_z 336
>
>     All the best,
>
>     David
>
>     The University of Edinburgh is a charitable body, registered in
>     Scotland, with registration number SC005336. Is e buidheann
>     carthannais a th’ ann an Oilthigh Dhùn Èideann, clàraichte an
>     Alba, àireamh clàraidh SC005336.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20211208/a29c8fd9/attachment-0001.html>