[petsc-users] HDF5 corruption

[Hapla Vaclav]

On 12 Oct 2021, at 11:56, Hapla Vaclav <vaclav.hapla at erdw.ethz.ch<mailto:vaclav.hapla at erdw.ethz.ch>> wrote:



On 8 Oct 2021, at 00:48, Jed Brown <jed at jedbrown.org<mailto:jed at jedbrown.org>> wrote:

Adrian Croucher <a.croucher at auckland.ac.nz<mailto:a.croucher at auckland.ac.nz>> writes:

hi Jed,

It looked to me like a call to h5f_flush() is all that is required.

Some people said there would be a performance hit (maybe ~ 10% slower),
which would be the trade-off for increased reliability. So if this were
made available via PetscViewerFlush(), I'd probably make it optional in
my code so the user could decide for themselves if it was worth it for them.

Do you think flushing would be a better option than closing/opening the
file between writes?

Yes, less costly at scale (metadata like opening files can be expensive on parallel file systems), and simpler to manage from your code.

I have just come across this a couple of days ago. I think PetscViewerFlush() [no-op for HDF5 currently] should call H5Fflush() for sure. I can do it now.

https://gitlab.com/petsc/petsc/-/merge_requests/4445


I agree with Jed that closing/opening can have significant overhead on large number of processes due to metadata processing.

Thanks,

Vaclav

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20211012/d43b1044/attachment-0001.html>