[petsc-users] Unable to create >4GB sized HDF5 files on Cray XC30

[Matthew Knepley]

On Sat, Aug 17, 2013 at 8:07 PM, Jed Brown <jedbrown at mcs.anl.gov> wrote:

> Juha Jäykkä <juhaj at iki.fi> writes:
>
> > Hi list!
> >
> > I wonder when and why (to make writing faster, I guess) this (and the
> > corresponding line in src/vec/vec/impls/mpi/pdvec.c) was added:
> >
> > petsc-3.3-p6/src/dm/impls/da/gr2.c:360:    status =
> H5Pset_chunk(chunkspace,
> > dim, chunkDims); CHKERRQ(status);
> >
> > I know that 3.3.6 is "old", but 3.4.2 performs the same call, so I'm not
> any
> > better off with that.
> >
> > This line causes problems, at least on a Cray XC30 machine, with HDF5
> library
> > telling me that "#010: H5Dchunk.c line 443 in H5D_chunk_construct():
> chunk
> > size must be < 4GB."
> >
> > I know chunked IO is good idea, but apparently PETSc uses disallowed
> > chunksizes.
>
> It doesn't make any sense to use a chunk size that is as large as the
> entire spatial domain.  As I understand it, H5Pset_chunk has the
> restriction that the chunk size cannot be larger than the data, so to be
> able to write small data files, PETSc has to set small chunk sizes for
> small data.
>
> I think this issue can be fixed by changing
>
>   chunkDims[dim] = dims[dim];
>
> to something like
>
>   chunkDims[dim] = PetscMin(dims[dim],dim_for_norminal_chunk_size);
>
> Does anyone have data on what chunk sizes would make a good default? 10
> MiB?
>

This was not a performance optimization. IIRC, I did this so we could leave
the time
domain of unspecified length, and write a chunk of values at each timestep.

   Matt

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20130818/e3f47d84/attachment.html>