[petsc-users] Problem with PETSc + HDF5 VecView

Tue Nov 25 15:40:53 CST 2014

On Tue, Nov 25, 2014 at 2:34 PM, Håkon Strandenes <haakon at hakostra.net>
wrote:

> I have done a whole lot of debugging on this issue. In my first post I
> wrote that "The VecLoad problems can rest for now, but the VecView are more
> serious.". I now believe that these problems in essence are the same. I
> have however not yet found out where the problem is, in PETSc, HDF5, MPT or
> (perhaps unlikely) the Lustre file system.
>
> The VecLoad bug is as follows: I have a properly formatted HDF5-file with
> data corresponding to a vector on a DMDA grid. If I try to load this data
> from the HDF5-file and into a Vec, it works with no error messages. The
> data is however severely corrupted. This is shown in the attached figure
> fig-COLELCTIVE.png, where the left column of figures is the three
> components of a vector that actually is in the HDF5 file, while the right
> column is how PETSc's VecLoad() reads the data. This bug does not cause any
> error messages what so ever, and is again dependent on the decomposition
> pattern.
>
> Another strange phenomena related to VecLoad() is that the occurrence of
> this bug seems to be dependent on the chunking in the dataset being loaded.
> Different chinking patterns seems to produce different results. This might
> be an important lead.
>
> My workaround is to replace the two occurrences of
>   H5Pset_dxpl_mpio(plist_id, H5FD_MPIO_COLLECTIVE)
> with
>   H5Pset_dxpl_mpio(plist_id, H5FD_MPIO_INDEPENDENT)
> in file src/dm/impls/da/gr2.c These two occurrences are found in the
> functions VecView_MPI_HDF5_DA() and VecLoad_HDF5_DA().
>
> This solves all problems (at least so far). Both VecLoad() and VecView()
> works as expected. See attached figure fig-INDEPENDENT.png. The left column
> is again input and the right column is output. I have actually made a small
> example program that performs two tasks: loads a Vec from a HDF5-file and
> writes the same Vec back again to a new file. When comparing the data
> inside these files, it is exactly identical for all combinations of
> decomposition, number of processes and chunking patterns in the input file
> I have tried.
>
> This leads me to an important question I cannot answer because I don't
> have enough knowledge and insight in HDF5: Is PETSc using the
> collective/independent flags correctly? Can you use collective IO in all
> cases for all DMDA's and all variants of decomposition?
>
> The following document describe some features of parallel HDF5:
> http://www.hdfgroup.org/HDF5/PHDF5/parallelhdf5hints.pdf
>
> Does PETSc currently satisfy the conditions for using collective IO and
> chunking as described on page 5-6? Perhaps the reason why this is not
> occurring on my workstation is that HDF5 recognise that my file system is
> serial and falls back to some simple serial IO routines? My cluster is
> equipped with a parallel Lustre file system, and as far as I know HDF5
> handles these differently. The same document also mentions that HDF5 in
> deed does create some internal datatypes to accomplish collective IO on
> chunked storage (top of page 6), as Matthew suggested.
>
> Any comments?
>

First, this is great debugging.

Second, my reading of the HDF5 document you linked to says that either
selection should be valid:

  "For non-regular hyperslab selection, parallel HDF5 uses independent IO
internally for this option."

so it ought to fall back to the INDEPENDENT model if it can't do collective
calls correctly. However,
it appears that the collective call has bugs.

My conclusion: Since you have determined that changing the setting to
INDEPENDENT produces
correct input/output in all the test cases, and since my understanding of
the HDF5 documentation is
that we should always be able to use COLLECTIVE as an option, this is an
HDF5 or MPT bug.

Does anyone else see the HDF5 differently? Also, it really looks to me like
HDF5 messed up the MPI
data type in the COLLECTIVE picture below, since it appears to be sliced
incorrectly.

Possible Remedies:

  1) We can allow you to turn off H5Pset_dxpl_mpio()

  2) Send this test case to the MPI/IO people at ANL

If you think 1) is what you want, we can do it. If you can package this
work for 2), it would be really valuable.

  Thanks,

    Matt

Tanks for your time.
>
> Best regards,
> Håkon Strandenes
>
>
> On 25. nov. 2014 18:47, Matthew Knepley wrote:
>
>> On Mon, Nov 24, 2014 at 1:10 PM, Håkon Strandenes <haakon at hakostra.net
>> <mailto:haakon at hakostra.net>> wrote:
>>
>>     Hi,
>>
>>     I have some problems with PETSc and HDF5 VecLoad/VecView. The
>>     VecLoad problems can rest for now, but the VecView are more serious.
>>
>>     In short: I have a 3D DMDA with and some vectors that I want to save
>>     to a HDF5 file. This works perfectly on my workstation, but not on
>>     the compute cluster I have access to. I have attached a typical
>>     error message.
>>
>>     I have also attached an piece of code that can trigger the error.
>>     The code is merely a 2D->3D rewrite of DMDA ex 10
>>     (http://www.mcs.anl.gov/petsc/__petsc-current/src/dm/
>> examples/__tutorials/ex10.c.html
>>     <http://www.mcs.anl.gov/petsc/petsc-current/src/dm/examples/
>> tutorials/ex10.c.html>),
>>
>>     nothing else is done.
>>
>>     The program typically works on small number of processes. I have
>>     successfully executed the attached program on up to 32 processes.
>>     That works. Always. I have never had a single success when trying to
>>     run on 64 processes. Always same error.
>>
>>     The computer I am struggling with is an SGI machine with SLES 11sp1
>>     and Intel CPUs, hence I have used Intels compilers. I have tried
>>     both 2013, 2014 and 2015 versions of the compilers, so that's
>>     probably not the cause. I have also tried GCC 4.9.1, just to be
>>     safe, same error there. The same compiler is used for both HDF5 and
>>     PETSc. The same error message occurs for both debug and release
>>     builds. I have tried HDF5 versions 1.8.11 and 1.8.13. I have tried
>>     PETSc version 3.4.1 and the latest from Git. The MPI implementation
>>     on the machine is SGI's MPT, and i have tried both 2.06 and 2.10.
>>     Always same error. Other MPI implementations is unfortunately not
>>     available.
>>
>>     What really drives me mad is that this works like a charm on my
>>     workstation with Linux Mint... I have successfully executed the
>>     attached example on 254 processes (my machine breaks down if I try
>>     anything more than that).
>>
>>     Does any of you have any tips on how to attack this problem and find
>>     out what's wrong?
>>
>>
>> This does sound like a pain to track down. It seems to be complaining
>> about an MPI datatype:
>>
>> #005: H5Dmpio.c line 998 in H5D__link_chunk_collective_io():
>> MPI_Type_struct failed
>>      major: Internal error (too specific to document in detail)
>>      minor: Some MPI function failed
>>    #006: H5Dmpio.c line 998 in H5D__link_chunk_collective_io(): Invalid
>> datatype argument
>>      major: Internal error (too specific to document in detail)
>>      minor: MPI Error String
>>
>> In this call, we pass in 'scalartype', which is H5T_NATIVE_DOUBLE
>> (unless you configured for
>> single precision). This was used successfully to create the dataspace,
>> so it is unlikely to be
>> the problem. I am guessing that HDF5 creates internal MPI datatypes to
>> use in the MPI/IO
>> routines (maybe using MPI_Type_struct).
>>
>> I believe we have seen type creation routines fail in some MPI
>> implementations if you try to
>> create too many of them. Right now, this looks a lot like a bug in MPT,
>> although it might be
>> an HDF5 bug with forgetting to release MPI types that they do not need.
>>
>>    Thanks,
>>
>>      Matt
>>
>>     Regards,
>>     Håkon Strandenes
>>
>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which
>> their experiments lead.
>> -- Norbert Wiener
>>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20141125/4df441a2/attachment.html>