[petsc-users] How to read/write a HDF5 file using petsc4py ?

Matthew Knepley knepley at gmail.com
Tue Dec 7 12:16:30 CST 2021 

On Tue, Dec 7, 2021 at 9:43 AM Quentin Chevalier <
quentin.chevalier at polytechnique.edu> wrote:

> @Matthew, as stated before, error output is unchanged, i.e.the python
> command below produces the same traceback :
>
> # python3 -c "from petsc4py import PETSc;
> PETSc.Viewer().createHDF5('d.h5')"
> Traceback (most recent call last):
>   File "<string>", line 1, in <module>
>   File "PETSc/Viewer.pyx", line 182, in petsc4py.PETSc.Viewer.createHDF5
> petsc4py.PETSc.Error: error code 86
> [0] PetscViewerSetType() at
> /usr/local/petsc/src/sys/classes/viewer/interface/viewreg.c:442
> [0] Unknown type. Check for miss-spelling or missing package:
> https://petsc.org/release/install/install/#external-packages
> [0] Unknown PetscViewer type given: hdf5
>

The reason I wanted the output was that the C output shows the configure
options that the PETSc library
was built with, However, Python seems to be eating this, so I cannot check.

It seems like using this container is counter-productive. If it was built
correctly, making these changes would be trivial.
Send mail to FEniCS (I am guessing Chris Richardson maintains this), and
ask how they intend people to change these
options.

  Thanks,

     Matt.


> @Wence that makes sense. I'd assumed that the original PETSc had been
> overwritten, and if the linking has gone wrong I'm surprised anything
> happens with petsc4py at all.
>
> Your tentative command gave :
>
> ERROR: Invalid requirement: '/usr/local/petsc/src/binding/petsc4py'
> Hint: It looks like a path. File
> '/usr/local/petsc/src/binding/petsc4py' does not exist.
>
> So I tested that global variables PETSC_ARCH & PETSC_DIR were correct
> then ran "pip install petsc4py" to restart petsc4py from scratch. This
> gives rise to a different error :
> # python3 -c "from petsc4py import PETSc"
> Traceback (most recent call last):
>   File "<string>", line 1, in <module>
>   File "/usr/local/lib/python3.9/dist-packages/petsc4py/PETSc.py",
> line 3, in <module>
>     PETSc = ImportPETSc(ARCH)
>   File "/usr/local/lib/python3.9/dist-packages/petsc4py/lib/__init__.py",
> line 29, in ImportPETSc
>     return Import('petsc4py', 'PETSc', path, arch)
>   File "/usr/local/lib/python3.9/dist-packages/petsc4py/lib/__init__.py",
> line 73, in Import
>     module = import_module(pkg, name, path, arch)
>   File "/usr/local/lib/python3.9/dist-packages/petsc4py/lib/__init__.py",
> line 58, in import_module
>     with f: return imp.load_module(fullname, f, fn, info)
>   File "/usr/lib/python3.9/imp.py", line 242, in load_module
>     return load_dynamic(name, filename, file)
>   File "/usr/lib/python3.9/imp.py", line 342, in load_dynamic
>     return _load(spec)
> ImportError:
> /usr/local/lib/python3.9/dist-packages/petsc4py/lib/linux-gnu-real-32/
> PETSc.cpython-39-x86_64-linux-gnu.so:
> undefined symbol: petscstack
>
> Not sure that it a step forward ; looks like petsc4py is broken now.
>
> Quentin
>
> On Tue, 7 Dec 2021 at 14:58, Matthew Knepley <knepley at gmail.com> wrote:
> >
> > On Tue, Dec 7, 2021 at 8:26 AM Quentin Chevalier <
> quentin.chevalier at polytechnique.edu> wrote:
> >>
> >> Ok my bad, that log corresponded to a tentative --download-hdf5. This
> >> log corresponds to the commands given above and has --with-hdf5 in its
> >> options.
> >
> >
> > Okay, this configure was successful and found HDF5
> >
> >>
> >> The whole process still results in the same error.
> >
> >
> > Now send me the complete error output with this PETSc.
> >
> >   Thanks,
> >
> >      Matt
> >
> >>
> >> Quentin
> >>
> >>
> >>
> >> Quentin CHEVALIER – IA parcours recherche
> >>
> >> LadHyX - Ecole polytechnique
> >>
> >> __________
> >>
> >>
> >>
> >> On Tue, 7 Dec 2021 at 13:59, Matthew Knepley <knepley at gmail.com> wrote:
> >> >
> >> > On Tue, Dec 7, 2021 at 3:55 AM Quentin Chevalier <
> quentin.chevalier at polytechnique.edu> wrote:
> >> >>
> >> >> Hello Matthew,
> >> >>
> >> >> That would indeed make sense.
> >> >>
> >> >> Full log is attached, I grepped hdf5 in there and didn't find
> anything alarming.
> >> >
> >> >
> >> > At the top of this log:
> >> >
> >> > Configure Options: --configModules=PETSc.Configure
> --optionsModule=config.compilerOptions PETSC_ARCH=linux-gnu-complex-64
> --COPTFLAGS=-O2 --CXXOPTFLAGS=-O2 --FOPTFLAGS=-O2 --with-make-np=2
> --with-64-bit-indices=yes --with-debugging=no --with-fortran-bindings=no
> --with-shared-libraries --download-hypre --download-mumps
> --download-ptscotch --download-scalapack --download-suitesparse
> --download-superlu_dist --with-scalar-type=complex
> >> >
> >> >
> >> > So the HDF5 option is not being specified.
> >> >
> >> >   Thanks,
> >> >
> >> >      Matt
> >> >
> >> >> Cheers,
> >> >>
> >> >> Quentin
> >> >>
> >> >>
> >> >>
> >> >>
> >> >> Quentin CHEVALIER – IA parcours recherche
> >> >>
> >> >> LadHyX - Ecole polytechnique
> >> >>
> >> >> __________
> >> >>
> >> >>
> >> >>
> >> >> On Mon, 6 Dec 2021 at 21:39, Matthew Knepley <knepley at gmail.com>
> wrote:
> >> >>>
> >> >>> On Mon, Dec 6, 2021 at 3:27 PM Quentin Chevalier <
> quentin.chevalier at polytechnique.edu> wrote:
> >> >>>>
> >> >>>> Fine. MWE is unchanged :
> >> >>>> * Run this docker container
> >> >>>> * Do : python3 -c "from petsc4py import PETSc;
> PETSc.Viewer().createHDF5('dummy.h5')"
> >> >>>>
> >> >>>> Updated attempt at a fix :
> >> >>>> * cd /usr/local/petsc/
> >> >>>> * ./configure PETSC_ARCH= linux-gnu-real-32
> PETSC_DIR=/usr/local/petsc --with-hdf5 --force
> >> >>>
> >> >>>
> >> >>> Did it find HDF5? If not, it will shut it off. You need to send us
> >> >>>
> >> >>>   $PETSC_DIR/configure.log
> >> >>>
> >> >>> so we can see what happened in the configure run.
> >> >>>
> >> >>>   Thanks,
> >> >>>
> >> >>>       Matt
> >> >>>
> >> >>>>
> >> >>>> * make PETSC_DIR=/usr/local/petsc PETSC-ARCH= linux-gnu-real-32 all
> >> >>>>
> >> >>>> Still no joy. The same error remains.
> >> >>>>
> >> >>>> Quentin
> >> >>>>
> >> >>>>
> >> >>>>
> >> >>>> On Mon, 6 Dec 2021 at 20:04, Pierre Jolivet <pierre at joliv.et>
> wrote:
> >> >>>> >
> >> >>>> >
> >> >>>> >
> >> >>>> > On 6 Dec 2021, at 7:42 PM, Quentin Chevalier <
> quentin.chevalier at polytechnique.edu> wrote:
> >> >>>> >
> >> >>>> > The PETSC_DIR exactly corresponds to the previous one, so I
> guess that rules option b) out, except if a specific option is required to
> overwrite a previous installation of PETSc. As for a), well I thought
> reconfiguring pretty direct, you're welcome to give me a hint as to what
> could be wrong in the following process.
> >> >>>> >
> >> >>>> > Steps to reproduce this behaviour are as follows :
> >> >>>> > * Run this docker container
> >> >>>> > * Do : python3 -c "from petsc4py import PETSc;
> PETSc.Viewer().createHDF5('dummy.h5')"
> >> >>>> >
> >> >>>> > After you get the error Unknown PetscViewer type, feel free to
> try :
> >> >>>> >
> >> >>>> > * cd /usr/local/petsc/
> >> >>>> > * ./configure --with-hfd5
> >> >>>> >
> >> >>>> >
> >> >>>> > It’s hdf5, not hfd5.
> >> >>>> > It’s PETSC_ARCH, not PETSC-ARCH.
> >> >>>> > PETSC_ARCH is missing from your configure line.
> >> >>>> >
> >> >>>> > Thanks,
> >> >>>> > Pierre
> >> >>>> >
> >> >>>> > * make PETSC_DIR=/usr/local/petsc PETSC-ARCH=linux-gnu-real-32
> all
> >> >>>> >
> >> >>>> > Then repeat the MWE and observe absolutely no behavioural change
> whatsoever. I'm afraid I don't know PETSc well enough to be surprised by
> that.
> >> >>>> >
> >> >>>> > Quentin
> >> >>>> >
> >> >>>> >
> >> >>>> >
> >> >>>> > Quentin CHEVALIER – IA parcours recherche
> >> >>>> >
> >> >>>> > LadHyX - Ecole polytechnique
> >> >>>> >
> >> >>>> > __________
> >> >>>> >
> >> >>>> >
> >> >>>> >
> >> >>>> > On Mon, 6 Dec 2021 at 19:24, Matthew Knepley <knepley at gmail.com>
> wrote:
> >> >>>> >>
> >> >>>> >> On Mon, Dec 6, 2021 at 1:22 PM Quentin Chevalier <
> quentin.chevalier at polytechnique.edu> wrote:
> >> >>>> >>>
> >> >>>> >>> It failed all of the tests included in `make
> >> >>>> >>> PETSC_DIR=/usr/local/petsc PETSC-ARCH=linux-gnu-real-32
> check`, with
> >> >>>> >>> the error `/usr/bin/bash: line 1: cd: src/snes/tutorials: No
> such file
> >> >>>> >>> or directory`
> >> >>>> >>>
> >> >>>> >>> I am therefore fairly confident this a "file absence" problem,
> and not
> >> >>>> >>> a compilation problem.
> >> >>>> >>>
> >> >>>> >>> I repeat that there was no error at compilation stage. The
> final stage
> >> >>>> >>> did present `gmake[3]: Nothing to be done for 'libs'.` but
> that's all.
> >> >>>> >>>
> >> >>>> >>> Again, running `./configure --with-hdf5` followed by a `make
> >> >>>> >>> PETSC_DIR=/usr/local/petsc PETSC-ARCH=linux-gnu-real-32 all`
> does not
> >> >>>> >>> change the problem. I get the same error at the same position
> as
> >> >>>> >>> before.
> >> >>>> >>
> >> >>>> >>
> >> >>>> >> If you reconfigured and rebuilt, it is impossible to get the
> same error, so
> >> >>>> >>
> >> >>>> >>   a) You did not reconfigure
> >> >>>> >>
> >> >>>> >>   b) Your new build is somewhere else on the machine
> >> >>>> >>
> >> >>>> >>   Thanks,
> >> >>>> >>
> >> >>>> >>      Matt
> >> >>>> >>
> >> >>>> >>>
> >> >>>> >>> I will comment I am running on OpenSUSE.
> >> >>>> >>>
> >> >>>> >>> Quentin
> >> >>>> >>>
> >> >>>> >>> On Mon, 6 Dec 2021 at 19:09, Matthew Knepley <
> knepley at gmail.com> wrote:
> >> >>>> >>> >
> >> >>>> >>> > On Mon, Dec 6, 2021 at 1:08 PM Quentin Chevalier <
> quentin.chevalier at polytechnique.edu> wrote:
> >> >>>> >>> >>
> >> >>>> >>> >> Hello Matthew and thanks for your quick response.
> >> >>>> >>> >>
> >> >>>> >>> >> I'm afraid I did try to snoop around the container and
> rerun PETSc's
> >> >>>> >>> >> configure with the --with-hdf5 option, to absolutely no
> avail.
> >> >>>> >>> >>
> >> >>>> >>> >> I didn't see any errors during config or make, but it
> failed the tests
> >> >>>> >>> >> (which aren't included in the minimal container I suppose)
> >> >>>> >>> >
> >> >>>> >>> >
> >> >>>> >>> > Failed which tests? What was the error?
> >> >>>> >>> >
> >> >>>> >>> >   Thanks,
> >> >>>> >>> >
> >> >>>> >>> >     Matt
> >> >>>> >>> >
> >> >>>> >>> >>
> >> >>>> >>> >> Quentin
> >> >>>> >>> >>
> >> >>>> >>> >>
> >> >>>> >>> >>
> >> >>>> >>> >> Quentin CHEVALIER – IA parcours recherche
> >> >>>> >>> >>
> >> >>>> >>> >> LadHyX - Ecole polytechnique
> >> >>>> >>> >>
> >> >>>> >>> >> __________
> >> >>>> >>> >>
> >> >>>> >>> >>
> >> >>>> >>> >>
> >> >>>> >>> >> On Mon, 6 Dec 2021 at 19:02, Matthew Knepley <
> knepley at gmail.com> wrote:
> >> >>>> >>> >> >
> >> >>>> >>> >> > On Mon, Dec 6, 2021 at 11:28 AM Quentin Chevalier <
> quentin.chevalier at polytechnique.edu> wrote:
> >> >>>> >>> >> >>
> >> >>>> >>> >> >> Hello PETSc users,
> >> >>>> >>> >> >>
> >> >>>> >>> >> >> This email is a duplicata of this gitlab issue, sorry
> for any inconvenience caused.
> >> >>>> >>> >> >>
> >> >>>> >>> >> >> I want to compute a PETSc vector in real mode, than
> perform calculations with it in complex mode. I want as much of this
> process to be parallel as possible. Right now, I compile PETSc in real
> mode, compute my vector and save it to a file, then switch to complex mode,
> read it, and move on.
> >> >>>> >>> >> >>
> >> >>>> >>> >> >> This creates unexpected behaviour using MPIIO, so on
> Lisandro Dalcinl's advice I'm moving to HDF5 format. My code is as follows
> (taking inspiration from petsc4py doc, a bitbucket example and another one,
> all top Google results for 'petsc hdf5') :
> >> >>>> >>> >> >>>
> >> >>>> >>> >> >>> viewer = PETSc.Viewer().createHDF5(file_name, 'r',
> COMM_WORLD)
> >> >>>> >>> >> >>> q.load(viewer)
> >> >>>> >>> >> >>> q.ghostUpdate(addv=PETSc.InsertMode.INSERT,
> mode=PETSc.ScatterMode.FORWARD)
> >> >>>> >>> >> >>
> >> >>>> >>> >> >>
> >> >>>> >>> >> >> This crashes my code. I obtain traceback :
> >> >>>> >>> >> >>>
> >> >>>> >>> >> >>>   File "/home/shared/code.py", line 121, in Load
> >> >>>> >>> >> >>>     viewer = PETSc.Viewer().createHDF5(file_name, 'r',
> COMM_WORLD)
> >> >>>> >>> >> >>>   File "PETSc/Viewer.pyx", line 182, in
> petsc4py.PETSc.Viewer.createHDF5
> >> >>>> >>> >> >>> petsc4py.PETSc.Error: error code 86
> >> >>>> >>> >> >>> [0] PetscViewerSetType() at
> /usr/local/petsc/src/sys/classes/viewer/interface/viewreg.c:442
> >> >>>> >>> >> >>> [0] Unknown type. Check for miss-spelling or missing
> package: https://petsc.org/release/install/install/#external-packages
> >> >>>> >>> >> >>> [0] Unknown PetscViewer type given: hdf5
> >> >>>> >>> >> >
> >> >>>> >>> >> > This means that PETSc has not been configured with HDF5
> (--with-hdf5 or --download-hdf5), so the container should be updated.
> >> >>>> >>> >> >
> >> >>>> >>> >> >   THanks,
> >> >>>> >>> >> >
> >> >>>> >>> >> >     Matt
> >> >>>> >>> >> >
> >> >>>> >>> >> >>
> >> >>>> >>> >> >> I have petsc4py 3.16 from this docker container (list of
> dependencies include PETSc and petsc4py).
> >> >>>> >>> >> >>
> >> >>>> >>> >> >> I'm pretty sure this is not intended behaviour. Any
> insight as to how to fix this issue (I tried running ./configure
> --with-hdf5 to no avail) or more generally to perform this jiggling between
> real and complex would be much appreciated,
> >> >>>> >>> >> >>
> >> >>>> >>> >> >> Kind regards.
> >> >>>> >>> >> >>
> >> >>>> >>> >> >> Quentin
> >> >>>> >>> >> >
> >> >>>> >>> >> >
> >> >>>> >>> >> >
> >> >>>> >>> >> > --
> >> >>>> >>> >> > What most experimenters take for granted before they
> begin their experiments is infinitely more interesting than any results to
> which their experiments lead.
> >> >>>> >>> >> > -- Norbert Wiener
> >> >>>> >>> >> >
> >> >>>> >>> >> > https://www.cse.buffalo.edu/~knepley/
> >> >>>> >>> >
> >> >>>> >>> >
> >> >>>> >>> >
> >> >>>> >>> > --
> >> >>>> >>> > What most experimenters take for granted before they begin
> their experiments is infinitely more interesting than any results to which
> their experiments lead.
> >> >>>> >>> > -- Norbert Wiener
> >> >>>> >>> >
> >> >>>> >>> > https://www.cse.buffalo.edu/~knepley/
> >> >>>> >>
> >> >>>> >>
> >> >>>> >>
> >> >>>> >> --
> >> >>>> >> What most experimenters take for granted before they begin
> their experiments is infinitely more interesting than any results to which
> their experiments lead.
> >> >>>> >> -- Norbert Wiener
> >> >>>> >>
> >> >>>> >> https://www.cse.buffalo.edu/~knepley/
> >> >>>> >
> >> >>>> > <image003.jpg>
> >> >>>> >
> >> >>>> >
> >> >>>
> >> >>>
> >> >>>
> >> >>> --
> >> >>> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> >> >>> -- Norbert Wiener
> >> >>>
> >> >>> https://www.cse.buffalo.edu/~knepley/
> >> >
> >> >
> >> >
> >> > --
> >> > What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> >> > -- Norbert Wiener
> >> >
> >> > https://www.cse.buffalo.edu/~knepley/
> >
> >
> >
> > --
> > What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> > -- Norbert Wiener
> >
> > https://www.cse.buffalo.edu/~knepley/
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20211207/958ec29f/attachment-0001.html>

More information about the petsc-users mailing list