[petsc-users] Unable to create >4GB sized HDF5 files on Cray XC30

[Juha Jäykkä]

> similarly.  It would make a lot more difference for something like
> visualization where subsets of the data are read, typically with
> independent IO.

True. Fortunately, h5repack can change the chunking afterwards if necessary. 
Of course, for truly huge files that's going to take ages.

> > turning chunking completely off works too
> Are you sure?  Did you try writing a second time step?  The

Yes. No. I just solve an equation, so I'm not interested in saving steps. 
Sometimes when I do, I save them into separate files. I tried to save them as 
time-steps sometime around 3.1 or 3.2, but at that point there were only two 
possibilities: Read and Truncate, so adding another timestep was impossible. 
>From what I see, there now is an FILE_MODE_APPEND possibility, but it won't be 
worth it for me.

> > See above, but note also that there can at most be 64k chunks in the file,
> > so fixing the chunk size to 10 MiB means limiting file size to 640 GiB.
> Thanks for noticing this limit.  This might come from the 64k limit
> on attribute sizes.

You're welcome.

> Chunk size needs to be collective.  We could compute an average size
> From each subdomain, but can't just use the subdomain size.

Right. The general idea should still work the same way.

> We might want the chunk size to be smaller than 4GiB anyway to avoid
> out-of-memory problems for readers and writers.

I decided against making the OOM point because I came to the conclusion 4 GiB 
is not going to restrict any system where one would contemplate solving 
problems of such size: any cluster or "true" supercomputer will have so many 
cores / node that even a modest 1 GiB / core will give > 4 GiB total.

But I agree anyway: there's no point in insisting on such huge chunk size 
unless there is a huge performance benefit to gain. And even then it might 
well...

> I think the chunk size (or maximum chunk size) should be settable by the
> user.

to be explicitly set by the user by this option, so that first-time-users and 
other less savvy users do not get hurt by out-of-memory errors if they are 
unlucky enough to not have more memory on their HPC system.

Cheers,
Juha

-- 
		 -----------------------------------------------
		| Juha Jäykkä, juhaj at iki.fi			|
		| http://koti.kapsi.fi/~juhaj/			|
		 -----------------------------------------------
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: This is a digitally signed message part.
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20130818/73971e6a/attachment.pgp>