[petsc-users] Unable to create >4GB sized HDF5 files on Cray XC30

[Juha Jäykkä]

> It doesn't make any sense to use a chunk size that is as large as the
> entire spatial domain.  As I understand it, H5Pset_chunk has the

Indeed. And even if the data was 1 TB (in my problematic case it was just 15 
GB), a chunk size even approaching 4 GB is unlikely to be optimal: HDF5 will 
always read at least a chunk, so if you want to do anything with a small 
subset of the data, say just 10 MB, you'll end up reading 4 GB.

> restriction that the chunk size cannot be larger than the data, so to be
> able to write small data files, PETSc has to set small chunk sizes for
> small data.

For small files, chunking is probably not going to change performance in any 
significant manner, so one option could be to simply not chunk small files at 
all and then chunk big files "optimally" – whatever that means. HDFgroup seems 
to think that "the chunk size be approximately equal to the average expected 
size of the data block needed by the application." 
(http://www.hdfgroup.org/training/HDFtraining/UsersGuide/Perform.fm2.html) For 
more chunking stuff: 

In the case of PETSc I think that means not the WHOLE application, but one MPI 
rank (or perhaps one SMP host running a mixture of MPI ranks and OpenMP 
threads), which is probably always going to be < 4 GB (except perhaps in the 
mixture case).

> I think this issue can be fixed by changing
> 
>   chunkDims[dim] = dims[dim];
> 
> to something like
> 
>   chunkDims[dim] = PetscMin(dims[dim],dim_for_norminal_chunk_size);

I'll see how that affects the performance in my case: turning chunking 
completely off works too, but I would not expect that to excel in performance. 

> Does anyone have data on what chunk sizes would make a good default? 10
> MiB?

See above, but note also that there can at most be 64k chunks in the file, so 
fixing the chunk size to 10 MiB means limiting file size to 640 GiB.

My suggestion is to give PETSc a little more logic here, something like this:

if sizeof(data) > 4GiB * 64k: no chunking # impossible to chunk!
elif sizeof(data) < small_file_limit: no chunking # probably best for speed
elif current rank's data size < 4 GB: chunk using current ranks data size
else divide current rank's data size by 2**(number of dimensions) until < 4 GB 
and then use that chunk size.

It is obvious what dimensions the chunks would be in that case. I hope I 
covered all contingencies there.

Cheers,
Juha

-- 
		 -----------------------------------------------
		| Juha Jäykkä, juhaj at iki.fi			|
		| http://koti.kapsi.fi/~juhaj/			|
		 -----------------------------------------------
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: This is a digitally signed message part.
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20130818/562d2673/attachment-0001.pgp>