[petsc-users] How to read/write a HDF5 file using petsc4py ?

Quentin Chevalier quentin.chevalier at polytechnique.edu
Tue Dec 7 08:43:15 CST 2021 

@Matthew, as stated before, error output is unchanged, i.e.the python
command below produces the same traceback :

# python3 -c "from petsc4py import PETSc; PETSc.Viewer().createHDF5('d.h5')"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "PETSc/Viewer.pyx", line 182, in petsc4py.PETSc.Viewer.createHDF5
petsc4py.PETSc.Error: error code 86
[0] PetscViewerSetType() at
/usr/local/petsc/src/sys/classes/viewer/interface/viewreg.c:442
[0] Unknown type. Check for miss-spelling or missing package:
https://petsc.org/release/install/install/#external-packages
[0] Unknown PetscViewer type given: hdf5

@Wence that makes sense. I'd assumed that the original PETSc had been
overwritten, and if the linking has gone wrong I'm surprised anything
happens with petsc4py at all.

Your tentative command gave :

ERROR: Invalid requirement: '/usr/local/petsc/src/binding/petsc4py'
Hint: It looks like a path. File
'/usr/local/petsc/src/binding/petsc4py' does not exist.

So I tested that global variables PETSC_ARCH & PETSC_DIR were correct
then ran "pip install petsc4py" to restart petsc4py from scratch. This
gives rise to a different error :
# python3 -c "from petsc4py import PETSc"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/local/lib/python3.9/dist-packages/petsc4py/PETSc.py",
line 3, in <module>
    PETSc = ImportPETSc(ARCH)
  File "/usr/local/lib/python3.9/dist-packages/petsc4py/lib/__init__.py",
line 29, in ImportPETSc
    return Import('petsc4py', 'PETSc', path, arch)
  File "/usr/local/lib/python3.9/dist-packages/petsc4py/lib/__init__.py",
line 73, in Import
    module = import_module(pkg, name, path, arch)
  File "/usr/local/lib/python3.9/dist-packages/petsc4py/lib/__init__.py",
line 58, in import_module
    with f: return imp.load_module(fullname, f, fn, info)
  File "/usr/lib/python3.9/imp.py", line 242, in load_module
    return load_dynamic(name, filename, file)
  File "/usr/lib/python3.9/imp.py", line 342, in load_dynamic
    return _load(spec)
ImportError: /usr/local/lib/python3.9/dist-packages/petsc4py/lib/linux-gnu-real-32/PETSc.cpython-39-x86_64-linux-gnu.so:
undefined symbol: petscstack

Not sure that it a step forward ; looks like petsc4py is broken now.

Quentin

On Tue, 7 Dec 2021 at 14:58, Matthew Knepley <knepley at gmail.com> wrote:
>
> On Tue, Dec 7, 2021 at 8:26 AM Quentin Chevalier <quentin.chevalier at polytechnique.edu> wrote:
>>
>> Ok my bad, that log corresponded to a tentative --download-hdf5. This
>> log corresponds to the commands given above and has --with-hdf5 in its
>> options.
>
>
> Okay, this configure was successful and found HDF5
>
>>
>> The whole process still results in the same error.
>
>
> Now send me the complete error output with this PETSc.
>
>   Thanks,
>
>      Matt
>
>>
>> Quentin
>>
>>
>>
>> Quentin CHEVALIER – IA parcours recherche
>>
>> LadHyX - Ecole polytechnique
>>
>> __________
>>
>>
>>
>> On Tue, 7 Dec 2021 at 13:59, Matthew Knepley <knepley at gmail.com> wrote:
>> >
>> > On Tue, Dec 7, 2021 at 3:55 AM Quentin Chevalier <quentin.chevalier at polytechnique.edu> wrote:
>> >>
>> >> Hello Matthew,
>> >>
>> >> That would indeed make sense.
>> >>
>> >> Full log is attached, I grepped hdf5 in there and didn't find anything alarming.
>> >
>> >
>> > At the top of this log:
>> >
>> > Configure Options: --configModules=PETSc.Configure --optionsModule=config.compilerOptions PETSC_ARCH=linux-gnu-complex-64 --COPTFLAGS=-O2 --CXXOPTFLAGS=-O2 --FOPTFLAGS=-O2 --with-make-np=2 --with-64-bit-indices=yes --with-debugging=no --with-fortran-bindings=no --with-shared-libraries --download-hypre --download-mumps --download-ptscotch --download-scalapack --download-suitesparse --download-superlu_dist --with-scalar-type=complex
>> >
>> >
>> > So the HDF5 option is not being specified.
>> >
>> >   Thanks,
>> >
>> >      Matt
>> >
>> >> Cheers,
>> >>
>> >> Quentin
>> >>
>> >>
>> >>
>> >>
>> >> Quentin CHEVALIER – IA parcours recherche
>> >>
>> >> LadHyX - Ecole polytechnique
>> >>
>> >> __________
>> >>
>> >>
>> >>
>> >> On Mon, 6 Dec 2021 at 21:39, Matthew Knepley <knepley at gmail.com> wrote:
>> >>>
>> >>> On Mon, Dec 6, 2021 at 3:27 PM Quentin Chevalier <quentin.chevalier at polytechnique.edu> wrote:
>> >>>>
>> >>>> Fine. MWE is unchanged :
>> >>>> * Run this docker container
>> >>>> * Do : python3 -c "from petsc4py import PETSc; PETSc.Viewer().createHDF5('dummy.h5')"
>> >>>>
>> >>>> Updated attempt at a fix :
>> >>>> * cd /usr/local/petsc/
>> >>>> * ./configure PETSC_ARCH= linux-gnu-real-32  PETSC_DIR=/usr/local/petsc --with-hdf5 --force
>> >>>
>> >>>
>> >>> Did it find HDF5? If not, it will shut it off. You need to send us
>> >>>
>> >>>   $PETSC_DIR/configure.log
>> >>>
>> >>> so we can see what happened in the configure run.
>> >>>
>> >>>   Thanks,
>> >>>
>> >>>       Matt
>> >>>
>> >>>>
>> >>>> * make PETSC_DIR=/usr/local/petsc PETSC-ARCH= linux-gnu-real-32 all
>> >>>>
>> >>>> Still no joy. The same error remains.
>> >>>>
>> >>>> Quentin
>> >>>>
>> >>>>
>> >>>>
>> >>>> On Mon, 6 Dec 2021 at 20:04, Pierre Jolivet <pierre at joliv.et> wrote:
>> >>>> >
>> >>>> >
>> >>>> >
>> >>>> > On 6 Dec 2021, at 7:42 PM, Quentin Chevalier <quentin.chevalier at polytechnique.edu> wrote:
>> >>>> >
>> >>>> > The PETSC_DIR exactly corresponds to the previous one, so I guess that rules option b) out, except if a specific option is required to overwrite a previous installation of PETSc. As for a), well I thought reconfiguring pretty direct, you're welcome to give me a hint as to what could be wrong in the following process.
>> >>>> >
>> >>>> > Steps to reproduce this behaviour are as follows :
>> >>>> > * Run this docker container
>> >>>> > * Do : python3 -c "from petsc4py import PETSc; PETSc.Viewer().createHDF5('dummy.h5')"
>> >>>> >
>> >>>> > After you get the error Unknown PetscViewer type, feel free to try :
>> >>>> >
>> >>>> > * cd /usr/local/petsc/
>> >>>> > * ./configure --with-hfd5
>> >>>> >
>> >>>> >
>> >>>> > It’s hdf5, not hfd5.
>> >>>> > It’s PETSC_ARCH, not PETSC-ARCH.
>> >>>> > PETSC_ARCH is missing from your configure line.
>> >>>> >
>> >>>> > Thanks,
>> >>>> > Pierre
>> >>>> >
>> >>>> > * make PETSC_DIR=/usr/local/petsc PETSC-ARCH=linux-gnu-real-32 all
>> >>>> >
>> >>>> > Then repeat the MWE and observe absolutely no behavioural change whatsoever. I'm afraid I don't know PETSc well enough to be surprised by that.
>> >>>> >
>> >>>> > Quentin
>> >>>> >
>> >>>> >
>> >>>> >
>> >>>> > Quentin CHEVALIER – IA parcours recherche
>> >>>> >
>> >>>> > LadHyX - Ecole polytechnique
>> >>>> >
>> >>>> > __________
>> >>>> >
>> >>>> >
>> >>>> >
>> >>>> > On Mon, 6 Dec 2021 at 19:24, Matthew Knepley <knepley at gmail.com> wrote:
>> >>>> >>
>> >>>> >> On Mon, Dec 6, 2021 at 1:22 PM Quentin Chevalier <quentin.chevalier at polytechnique.edu> wrote:
>> >>>> >>>
>> >>>> >>> It failed all of the tests included in `make
>> >>>> >>> PETSC_DIR=/usr/local/petsc PETSC-ARCH=linux-gnu-real-32 check`, with
>> >>>> >>> the error `/usr/bin/bash: line 1: cd: src/snes/tutorials: No such file
>> >>>> >>> or directory`
>> >>>> >>>
>> >>>> >>> I am therefore fairly confident this a "file absence" problem, and not
>> >>>> >>> a compilation problem.
>> >>>> >>>
>> >>>> >>> I repeat that there was no error at compilation stage. The final stage
>> >>>> >>> did present `gmake[3]: Nothing to be done for 'libs'.` but that's all.
>> >>>> >>>
>> >>>> >>> Again, running `./configure --with-hdf5` followed by a `make
>> >>>> >>> PETSC_DIR=/usr/local/petsc PETSC-ARCH=linux-gnu-real-32 all` does not
>> >>>> >>> change the problem. I get the same error at the same position as
>> >>>> >>> before.
>> >>>> >>
>> >>>> >>
>> >>>> >> If you reconfigured and rebuilt, it is impossible to get the same error, so
>> >>>> >>
>> >>>> >>   a) You did not reconfigure
>> >>>> >>
>> >>>> >>   b) Your new build is somewhere else on the machine
>> >>>> >>
>> >>>> >>   Thanks,
>> >>>> >>
>> >>>> >>      Matt
>> >>>> >>
>> >>>> >>>
>> >>>> >>> I will comment I am running on OpenSUSE.
>> >>>> >>>
>> >>>> >>> Quentin
>> >>>> >>>
>> >>>> >>> On Mon, 6 Dec 2021 at 19:09, Matthew Knepley <knepley at gmail.com> wrote:
>> >>>> >>> >
>> >>>> >>> > On Mon, Dec 6, 2021 at 1:08 PM Quentin Chevalier <quentin.chevalier at polytechnique.edu> wrote:
>> >>>> >>> >>
>> >>>> >>> >> Hello Matthew and thanks for your quick response.
>> >>>> >>> >>
>> >>>> >>> >> I'm afraid I did try to snoop around the container and rerun PETSc's
>> >>>> >>> >> configure with the --with-hdf5 option, to absolutely no avail.
>> >>>> >>> >>
>> >>>> >>> >> I didn't see any errors during config or make, but it failed the tests
>> >>>> >>> >> (which aren't included in the minimal container I suppose)
>> >>>> >>> >
>> >>>> >>> >
>> >>>> >>> > Failed which tests? What was the error?
>> >>>> >>> >
>> >>>> >>> >   Thanks,
>> >>>> >>> >
>> >>>> >>> >     Matt
>> >>>> >>> >
>> >>>> >>> >>
>> >>>> >>> >> Quentin
>> >>>> >>> >>
>> >>>> >>> >>
>> >>>> >>> >>
>> >>>> >>> >> Quentin CHEVALIER – IA parcours recherche
>> >>>> >>> >>
>> >>>> >>> >> LadHyX - Ecole polytechnique
>> >>>> >>> >>
>> >>>> >>> >> __________
>> >>>> >>> >>
>> >>>> >>> >>
>> >>>> >>> >>
>> >>>> >>> >> On Mon, 6 Dec 2021 at 19:02, Matthew Knepley <knepley at gmail.com> wrote:
>> >>>> >>> >> >
>> >>>> >>> >> > On Mon, Dec 6, 2021 at 11:28 AM Quentin Chevalier <quentin.chevalier at polytechnique.edu> wrote:
>> >>>> >>> >> >>
>> >>>> >>> >> >> Hello PETSc users,
>> >>>> >>> >> >>
>> >>>> >>> >> >> This email is a duplicata of this gitlab issue, sorry for any inconvenience caused.
>> >>>> >>> >> >>
>> >>>> >>> >> >> I want to compute a PETSc vector in real mode, than perform calculations with it in complex mode. I want as much of this process to be parallel as possible. Right now, I compile PETSc in real mode, compute my vector and save it to a file, then switch to complex mode, read it, and move on.
>> >>>> >>> >> >>
>> >>>> >>> >> >> This creates unexpected behaviour using MPIIO, so on Lisandro Dalcinl's advice I'm moving to HDF5 format. My code is as follows (taking inspiration from petsc4py doc, a bitbucket example and another one, all top Google results for 'petsc hdf5') :
>> >>>> >>> >> >>>
>> >>>> >>> >> >>> viewer = PETSc.Viewer().createHDF5(file_name, 'r', COMM_WORLD)
>> >>>> >>> >> >>> q.load(viewer)
>> >>>> >>> >> >>> q.ghostUpdate(addv=PETSc.InsertMode.INSERT, mode=PETSc.ScatterMode.FORWARD)
>> >>>> >>> >> >>
>> >>>> >>> >> >>
>> >>>> >>> >> >> This crashes my code. I obtain traceback :
>> >>>> >>> >> >>>
>> >>>> >>> >> >>>   File "/home/shared/code.py", line 121, in Load
>> >>>> >>> >> >>>     viewer = PETSc.Viewer().createHDF5(file_name, 'r', COMM_WORLD)
>> >>>> >>> >> >>>   File "PETSc/Viewer.pyx", line 182, in petsc4py.PETSc.Viewer.createHDF5
>> >>>> >>> >> >>> petsc4py.PETSc.Error: error code 86
>> >>>> >>> >> >>> [0] PetscViewerSetType() at /usr/local/petsc/src/sys/classes/viewer/interface/viewreg.c:442
>> >>>> >>> >> >>> [0] Unknown type. Check for miss-spelling or missing package: https://petsc.org/release/install/install/#external-packages
>> >>>> >>> >> >>> [0] Unknown PetscViewer type given: hdf5
>> >>>> >>> >> >
>> >>>> >>> >> > This means that PETSc has not been configured with HDF5 (--with-hdf5 or --download-hdf5), so the container should be updated.
>> >>>> >>> >> >
>> >>>> >>> >> >   THanks,
>> >>>> >>> >> >
>> >>>> >>> >> >     Matt
>> >>>> >>> >> >
>> >>>> >>> >> >>
>> >>>> >>> >> >> I have petsc4py 3.16 from this docker container (list of dependencies include PETSc and petsc4py).
>> >>>> >>> >> >>
>> >>>> >>> >> >> I'm pretty sure this is not intended behaviour. Any insight as to how to fix this issue (I tried running ./configure --with-hdf5 to no avail) or more generally to perform this jiggling between real and complex would be much appreciated,
>> >>>> >>> >> >>
>> >>>> >>> >> >> Kind regards.
>> >>>> >>> >> >>
>> >>>> >>> >> >> Quentin
>> >>>> >>> >> >
>> >>>> >>> >> >
>> >>>> >>> >> >
>> >>>> >>> >> > --
>> >>>> >>> >> > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>> >>>> >>> >> > -- Norbert Wiener
>> >>>> >>> >> >
>> >>>> >>> >> > https://www.cse.buffalo.edu/~knepley/
>> >>>> >>> >
>> >>>> >>> >
>> >>>> >>> >
>> >>>> >>> > --
>> >>>> >>> > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>> >>>> >>> > -- Norbert Wiener
>> >>>> >>> >
>> >>>> >>> > https://www.cse.buffalo.edu/~knepley/
>> >>>> >>
>> >>>> >>
>> >>>> >>
>> >>>> >> --
>> >>>> >> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>> >>>> >> -- Norbert Wiener
>> >>>> >>
>> >>>> >> https://www.cse.buffalo.edu/~knepley/
>> >>>> >
>> >>>> > <image003.jpg>
>> >>>> >
>> >>>> >
>> >>>
>> >>>
>> >>>
>> >>> --
>> >>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>> >>> -- Norbert Wiener
>> >>>
>> >>> https://www.cse.buffalo.edu/~knepley/
>> >
>> >
>> >
>> > --
>> > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>> > -- Norbert Wiener
>> >
>> > https://www.cse.buffalo.edu/~knepley/
>
>
>
> --
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/

More information about the petsc-users mailing list