[petsc-users] Unable to create >4GB sized HDF5 files on Cray XC30

[Jed Brown]

Juha Jäykkä <juhaj at iki.fi> writes:

> Hi list!
>
> I wonder when and why (to make writing faster, I guess) this (and the 
> corresponding line in src/vec/vec/impls/mpi/pdvec.c) was added:
>
> petsc-3.3-p6/src/dm/impls/da/gr2.c:360:    status = H5Pset_chunk(chunkspace, 
> dim, chunkDims); CHKERRQ(status);
>
> I know that 3.3.6 is "old", but 3.4.2 performs the same call, so I'm not any 
> better off with that.
>
> This line causes problems, at least on a Cray XC30 machine, with HDF5 library 
> telling me that "#010: H5Dchunk.c line 443 in H5D_chunk_construct(): chunk 
> size must be < 4GB."
>
> I know chunked IO is good idea, but apparently PETSc uses disallowed 
> chunksizes.

It doesn't make any sense to use a chunk size that is as large as the
entire spatial domain.  As I understand it, H5Pset_chunk has the
restriction that the chunk size cannot be larger than the data, so to be
able to write small data files, PETSc has to set small chunk sizes for
small data.

I think this issue can be fixed by changing

  chunkDims[dim] = dims[dim];

to something like

  chunkDims[dim] = PetscMin(dims[dim],dim_for_norminal_chunk_size);

Does anyone have data on what chunk sizes would make a good default? 10
MiB?
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20130817/899944c2/attachment.pgp>