[petsc-users] How to read/write a HDF5 file using petsc4py ?
Matthew Knepley
knepley at gmail.com
Wed Dec 8 06:22:52 CST 2021
On Wed, Dec 8, 2021 at 4:08 AM Quentin Chevalier <
quentin.chevalier at polytechnique.edu> wrote:
> @all thanks for your time it's heartening to see a lively community.
>
> @Barry I've restarted the container and grabbed the .log file directly
> after the docker magic. I've tried a make check, it unsurprisingly spews
> the same answer as before :
>
> Running check examples to verify correct installation
> Using PETSC_DIR=/usr/local/petsc and PETSC_ARCH=linux-gnu-real-32
> /usr/bin/bash: line 1: cd: src/snes/tutorials: No such file or directory
> /usr/bin/bash: line 1: cd: src/snes/tutorials: No such file or directory
> gmake[3]: *** No rule to make target 'testex19'. Stop.
> gmake[2]: *** [makefile:155: check_build] Error 2
>
This happens if you run 'make check' without defining PETSC_DIR in your
environment, since we are including
makefiles with PETSC_DIR in the path and make does not allow proper error
messages in that case.
Thanks,
Matt
> @Matthew ok will do, but I think @Lawrence has already provided that
> answer. It's possible to change the dockerfile and recompute the dolfinx
> image with hdf5, only it is a time-consuming process.
>
> Quentin
>
>
>
> [image: cid:image003.jpg at 01D690CB.3B3FDC10]
>
> Quentin CHEVALIER – IA parcours recherche
>
> LadHyX - Ecole polytechnique
>
> __________
>
>
> On Tue, 7 Dec 2021 at 19:16, Matthew Knepley <knepley at gmail.com> wrote:
>
>> On Tue, Dec 7, 2021 at 9:43 AM Quentin Chevalier <
>> quentin.chevalier at polytechnique.edu> wrote:
>>
>>> @Matthew, as stated before, error output is unchanged, i.e.the python
>>> command below produces the same traceback :
>>>
>>> # python3 -c "from petsc4py import PETSc;
>>> PETSc.Viewer().createHDF5('d.h5')"
>>> Traceback (most recent call last):
>>> File "<string>", line 1, in <module>
>>> File "PETSc/Viewer.pyx", line 182, in petsc4py.PETSc.Viewer.createHDF5
>>> petsc4py.PETSc.Error: error code 86
>>> [0] PetscViewerSetType() at
>>> /usr/local/petsc/src/sys/classes/viewer/interface/viewreg.c:442
>>> [0] Unknown type. Check for miss-spelling or missing package:
>>> https://petsc.org/release/install/install/#external-packages
>>> [0] Unknown PetscViewer type given: hdf5
>>>
>>
>> The reason I wanted the output was that the C output shows the configure
>> options that the PETSc library
>> was built with, However, Python seems to be eating this, so I cannot
>> check.
>>
>> It seems like using this container is counter-productive. If it was built
>> correctly, making these changes would be trivial.
>> Send mail to FEniCS (I am guessing Chris Richardson maintains this), and
>> ask how they intend people to change these
>> options.
>>
>> Thanks,
>>
>> Matt.
>>
>>
>>> @Wence that makes sense. I'd assumed that the original PETSc had been
>>> overwritten, and if the linking has gone wrong I'm surprised anything
>>> happens with petsc4py at all.
>>>
>>> Your tentative command gave :
>>>
>>> ERROR: Invalid requirement: '/usr/local/petsc/src/binding/petsc4py'
>>> Hint: It looks like a path. File
>>> '/usr/local/petsc/src/binding/petsc4py' does not exist.
>>>
>>> So I tested that global variables PETSC_ARCH & PETSC_DIR were correct
>>> then ran "pip install petsc4py" to restart petsc4py from scratch. This
>>> gives rise to a different error :
>>> # python3 -c "from petsc4py import PETSc"
>>> Traceback (most recent call last):
>>> File "<string>", line 1, in <module>
>>> File "/usr/local/lib/python3.9/dist-packages/petsc4py/PETSc.py",
>>> line 3, in <module>
>>> PETSc = ImportPETSc(ARCH)
>>> File "/usr/local/lib/python3.9/dist-packages/petsc4py/lib/__init__.py",
>>> line 29, in ImportPETSc
>>> return Import('petsc4py', 'PETSc', path, arch)
>>> File "/usr/local/lib/python3.9/dist-packages/petsc4py/lib/__init__.py",
>>> line 73, in Import
>>> module = import_module(pkg, name, path, arch)
>>> File "/usr/local/lib/python3.9/dist-packages/petsc4py/lib/__init__.py",
>>> line 58, in import_module
>>> with f: return imp.load_module(fullname, f, fn, info)
>>> File "/usr/lib/python3.9/imp.py", line 242, in load_module
>>> return load_dynamic(name, filename, file)
>>> File "/usr/lib/python3.9/imp.py", line 342, in load_dynamic
>>> return _load(spec)
>>> ImportError:
>>> /usr/local/lib/python3.9/dist-packages/petsc4py/lib/linux-gnu-real-32/
>>> PETSc.cpython-39-x86_64-linux-gnu.so:
>>> undefined symbol: petscstack
>>>
>>> Not sure that it a step forward ; looks like petsc4py is broken now.
>>>
>>> Quentin
>>>
>>> On Tue, 7 Dec 2021 at 14:58, Matthew Knepley <knepley at gmail.com> wrote:
>>> >
>>> > On Tue, Dec 7, 2021 at 8:26 AM Quentin Chevalier <
>>> quentin.chevalier at polytechnique.edu> wrote:
>>> >>
>>> >> Ok my bad, that log corresponded to a tentative --download-hdf5. This
>>> >> log corresponds to the commands given above and has --with-hdf5 in its
>>> >> options.
>>> >
>>> >
>>> > Okay, this configure was successful and found HDF5
>>> >
>>> >>
>>> >> The whole process still results in the same error.
>>> >
>>> >
>>> > Now send me the complete error output with this PETSc.
>>> >
>>> > Thanks,
>>> >
>>> > Matt
>>> >
>>> >>
>>> >> Quentin
>>> >>
>>> >>
>>> >>
>>> >> Quentin CHEVALIER – IA parcours recherche
>>> >>
>>> >> LadHyX - Ecole polytechnique
>>> >>
>>> >> __________
>>> >>
>>> >>
>>> >>
>>> >> On Tue, 7 Dec 2021 at 13:59, Matthew Knepley <knepley at gmail.com>
>>> wrote:
>>> >> >
>>> >> > On Tue, Dec 7, 2021 at 3:55 AM Quentin Chevalier <
>>> quentin.chevalier at polytechnique.edu> wrote:
>>> >> >>
>>> >> >> Hello Matthew,
>>> >> >>
>>> >> >> That would indeed make sense.
>>> >> >>
>>> >> >> Full log is attached, I grepped hdf5 in there and didn't find
>>> anything alarming.
>>> >> >
>>> >> >
>>> >> > At the top of this log:
>>> >> >
>>> >> > Configure Options: --configModules=PETSc.Configure
>>> --optionsModule=config.compilerOptions PETSC_ARCH=linux-gnu-complex-64
>>> --COPTFLAGS=-O2 --CXXOPTFLAGS=-O2 --FOPTFLAGS=-O2 --with-make-np=2
>>> --with-64-bit-indices=yes --with-debugging=no --with-fortran-bindings=no
>>> --with-shared-libraries --download-hypre --download-mumps
>>> --download-ptscotch --download-scalapack --download-suitesparse
>>> --download-superlu_dist --with-scalar-type=complex
>>> >> >
>>> >> >
>>> >> > So the HDF5 option is not being specified.
>>> >> >
>>> >> > Thanks,
>>> >> >
>>> >> > Matt
>>> >> >
>>> >> >> Cheers,
>>> >> >>
>>> >> >> Quentin
>>> >> >>
>>> >> >>
>>> >> >>
>>> >> >>
>>> >> >> Quentin CHEVALIER – IA parcours recherche
>>> >> >>
>>> >> >> LadHyX - Ecole polytechnique
>>> >> >>
>>> >> >> __________
>>> >> >>
>>> >> >>
>>> >> >>
>>> >> >> On Mon, 6 Dec 2021 at 21:39, Matthew Knepley <knepley at gmail.com>
>>> wrote:
>>> >> >>>
>>> >> >>> On Mon, Dec 6, 2021 at 3:27 PM Quentin Chevalier <
>>> quentin.chevalier at polytechnique.edu> wrote:
>>> >> >>>>
>>> >> >>>> Fine. MWE is unchanged :
>>> >> >>>> * Run this docker container
>>> >> >>>> * Do : python3 -c "from petsc4py import PETSc;
>>> PETSc.Viewer().createHDF5('dummy.h5')"
>>> >> >>>>
>>> >> >>>> Updated attempt at a fix :
>>> >> >>>> * cd /usr/local/petsc/
>>> >> >>>> * ./configure PETSC_ARCH= linux-gnu-real-32
>>> PETSC_DIR=/usr/local/petsc --with-hdf5 --force
>>> >> >>>
>>> >> >>>
>>> >> >>> Did it find HDF5? If not, it will shut it off. You need to send us
>>> >> >>>
>>> >> >>> $PETSC_DIR/configure.log
>>> >> >>>
>>> >> >>> so we can see what happened in the configure run.
>>> >> >>>
>>> >> >>> Thanks,
>>> >> >>>
>>> >> >>> Matt
>>> >> >>>
>>> >> >>>>
>>> >> >>>> * make PETSC_DIR=/usr/local/petsc PETSC-ARCH= linux-gnu-real-32
>>> all
>>> >> >>>>
>>> >> >>>> Still no joy. The same error remains.
>>> >> >>>>
>>> >> >>>> Quentin
>>> >> >>>>
>>> >> >>>>
>>> >> >>>>
>>> >> >>>> On Mon, 6 Dec 2021 at 20:04, Pierre Jolivet <pierre at joliv.et>
>>> wrote:
>>> >> >>>> >
>>> >> >>>> >
>>> >> >>>> >
>>> >> >>>> > On 6 Dec 2021, at 7:42 PM, Quentin Chevalier <
>>> quentin.chevalier at polytechnique.edu> wrote:
>>> >> >>>> >
>>> >> >>>> > The PETSC_DIR exactly corresponds to the previous one, so I
>>> guess that rules option b) out, except if a specific option is required to
>>> overwrite a previous installation of PETSc. As for a), well I thought
>>> reconfiguring pretty direct, you're welcome to give me a hint as to what
>>> could be wrong in the following process.
>>> >> >>>> >
>>> >> >>>> > Steps to reproduce this behaviour are as follows :
>>> >> >>>> > * Run this docker container
>>> >> >>>> > * Do : python3 -c "from petsc4py import PETSc;
>>> PETSc.Viewer().createHDF5('dummy.h5')"
>>> >> >>>> >
>>> >> >>>> > After you get the error Unknown PetscViewer type, feel free to
>>> try :
>>> >> >>>> >
>>> >> >>>> > * cd /usr/local/petsc/
>>> >> >>>> > * ./configure --with-hfd5
>>> >> >>>> >
>>> >> >>>> >
>>> >> >>>> > It’s hdf5, not hfd5.
>>> >> >>>> > It’s PETSC_ARCH, not PETSC-ARCH.
>>> >> >>>> > PETSC_ARCH is missing from your configure line.
>>> >> >>>> >
>>> >> >>>> > Thanks,
>>> >> >>>> > Pierre
>>> >> >>>> >
>>> >> >>>> > * make PETSC_DIR=/usr/local/petsc PETSC-ARCH=linux-gnu-real-32
>>> all
>>> >> >>>> >
>>> >> >>>> > Then repeat the MWE and observe absolutely no behavioural
>>> change whatsoever. I'm afraid I don't know PETSc well enough to be
>>> surprised by that.
>>> >> >>>> >
>>> >> >>>> > Quentin
>>> >> >>>> >
>>> >> >>>> >
>>> >> >>>> >
>>> >> >>>> > Quentin CHEVALIER – IA parcours recherche
>>> >> >>>> >
>>> >> >>>> > LadHyX - Ecole polytechnique
>>> >> >>>> >
>>> >> >>>> > __________
>>> >> >>>> >
>>> >> >>>> >
>>> >> >>>> >
>>> >> >>>> > On Mon, 6 Dec 2021 at 19:24, Matthew Knepley <
>>> knepley at gmail.com> wrote:
>>> >> >>>> >>
>>> >> >>>> >> On Mon, Dec 6, 2021 at 1:22 PM Quentin Chevalier <
>>> quentin.chevalier at polytechnique.edu> wrote:
>>> >> >>>> >>>
>>> >> >>>> >>> It failed all of the tests included in `make
>>> >> >>>> >>> PETSC_DIR=/usr/local/petsc PETSC-ARCH=linux-gnu-real-32
>>> check`, with
>>> >> >>>> >>> the error `/usr/bin/bash: line 1: cd: src/snes/tutorials: No
>>> such file
>>> >> >>>> >>> or directory`
>>> >> >>>> >>>
>>> >> >>>> >>> I am therefore fairly confident this a "file absence"
>>> problem, and not
>>> >> >>>> >>> a compilation problem.
>>> >> >>>> >>>
>>> >> >>>> >>> I repeat that there was no error at compilation stage. The
>>> final stage
>>> >> >>>> >>> did present `gmake[3]: Nothing to be done for 'libs'.` but
>>> that's all.
>>> >> >>>> >>>
>>> >> >>>> >>> Again, running `./configure --with-hdf5` followed by a `make
>>> >> >>>> >>> PETSC_DIR=/usr/local/petsc PETSC-ARCH=linux-gnu-real-32 all`
>>> does not
>>> >> >>>> >>> change the problem. I get the same error at the same
>>> position as
>>> >> >>>> >>> before.
>>> >> >>>> >>
>>> >> >>>> >>
>>> >> >>>> >> If you reconfigured and rebuilt, it is impossible to get the
>>> same error, so
>>> >> >>>> >>
>>> >> >>>> >> a) You did not reconfigure
>>> >> >>>> >>
>>> >> >>>> >> b) Your new build is somewhere else on the machine
>>> >> >>>> >>
>>> >> >>>> >> Thanks,
>>> >> >>>> >>
>>> >> >>>> >> Matt
>>> >> >>>> >>
>>> >> >>>> >>>
>>> >> >>>> >>> I will comment I am running on OpenSUSE.
>>> >> >>>> >>>
>>> >> >>>> >>> Quentin
>>> >> >>>> >>>
>>> >> >>>> >>> On Mon, 6 Dec 2021 at 19:09, Matthew Knepley <
>>> knepley at gmail.com> wrote:
>>> >> >>>> >>> >
>>> >> >>>> >>> > On Mon, Dec 6, 2021 at 1:08 PM Quentin Chevalier <
>>> quentin.chevalier at polytechnique.edu> wrote:
>>> >> >>>> >>> >>
>>> >> >>>> >>> >> Hello Matthew and thanks for your quick response.
>>> >> >>>> >>> >>
>>> >> >>>> >>> >> I'm afraid I did try to snoop around the container and
>>> rerun PETSc's
>>> >> >>>> >>> >> configure with the --with-hdf5 option, to absolutely no
>>> avail.
>>> >> >>>> >>> >>
>>> >> >>>> >>> >> I didn't see any errors during config or make, but it
>>> failed the tests
>>> >> >>>> >>> >> (which aren't included in the minimal container I suppose)
>>> >> >>>> >>> >
>>> >> >>>> >>> >
>>> >> >>>> >>> > Failed which tests? What was the error?
>>> >> >>>> >>> >
>>> >> >>>> >>> > Thanks,
>>> >> >>>> >>> >
>>> >> >>>> >>> > Matt
>>> >> >>>> >>> >
>>> >> >>>> >>> >>
>>> >> >>>> >>> >> Quentin
>>> >> >>>> >>> >>
>>> >> >>>> >>> >>
>>> >> >>>> >>> >>
>>> >> >>>> >>> >> Quentin CHEVALIER – IA parcours recherche
>>> >> >>>> >>> >>
>>> >> >>>> >>> >> LadHyX - Ecole polytechnique
>>> >> >>>> >>> >>
>>> >> >>>> >>> >> __________
>>> >> >>>> >>> >>
>>> >> >>>> >>> >>
>>> >> >>>> >>> >>
>>> >> >>>> >>> >> On Mon, 6 Dec 2021 at 19:02, Matthew Knepley <
>>> knepley at gmail.com> wrote:
>>> >> >>>> >>> >> >
>>> >> >>>> >>> >> > On Mon, Dec 6, 2021 at 11:28 AM Quentin Chevalier <
>>> quentin.chevalier at polytechnique.edu> wrote:
>>> >> >>>> >>> >> >>
>>> >> >>>> >>> >> >> Hello PETSc users,
>>> >> >>>> >>> >> >>
>>> >> >>>> >>> >> >> This email is a duplicata of this gitlab issue, sorry
>>> for any inconvenience caused.
>>> >> >>>> >>> >> >>
>>> >> >>>> >>> >> >> I want to compute a PETSc vector in real mode, than
>>> perform calculations with it in complex mode. I want as much of this
>>> process to be parallel as possible. Right now, I compile PETSc in real
>>> mode, compute my vector and save it to a file, then switch to complex mode,
>>> read it, and move on.
>>> >> >>>> >>> >> >>
>>> >> >>>> >>> >> >> This creates unexpected behaviour using MPIIO, so on
>>> Lisandro Dalcinl's advice I'm moving to HDF5 format. My code is as follows
>>> (taking inspiration from petsc4py doc, a bitbucket example and another one,
>>> all top Google results for 'petsc hdf5') :
>>> >> >>>> >>> >> >>>
>>> >> >>>> >>> >> >>> viewer = PETSc.Viewer().createHDF5(file_name, 'r',
>>> COMM_WORLD)
>>> >> >>>> >>> >> >>> q.load(viewer)
>>> >> >>>> >>> >> >>> q.ghostUpdate(addv=PETSc.InsertMode.INSERT,
>>> mode=PETSc.ScatterMode.FORWARD)
>>> >> >>>> >>> >> >>
>>> >> >>>> >>> >> >>
>>> >> >>>> >>> >> >> This crashes my code. I obtain traceback :
>>> >> >>>> >>> >> >>>
>>> >> >>>> >>> >> >>> File "/home/shared/code.py", line 121, in Load
>>> >> >>>> >>> >> >>> viewer = PETSc.Viewer().createHDF5(file_name,
>>> 'r', COMM_WORLD)
>>> >> >>>> >>> >> >>> File "PETSc/Viewer.pyx", line 182, in
>>> petsc4py.PETSc.Viewer.createHDF5
>>> >> >>>> >>> >> >>> petsc4py.PETSc.Error: error code 86
>>> >> >>>> >>> >> >>> [0] PetscViewerSetType() at
>>> /usr/local/petsc/src/sys/classes/viewer/interface/viewreg.c:442
>>> >> >>>> >>> >> >>> [0] Unknown type. Check for miss-spelling or missing
>>> package: https://petsc.org/release/install/install/#external-packages
>>> >> >>>> >>> >> >>> [0] Unknown PetscViewer type given: hdf5
>>> >> >>>> >>> >> >
>>> >> >>>> >>> >> > This means that PETSc has not been configured with HDF5
>>> (--with-hdf5 or --download-hdf5), so the container should be updated.
>>> >> >>>> >>> >> >
>>> >> >>>> >>> >> > THanks,
>>> >> >>>> >>> >> >
>>> >> >>>> >>> >> > Matt
>>> >> >>>> >>> >> >
>>> >> >>>> >>> >> >>
>>> >> >>>> >>> >> >> I have petsc4py 3.16 from this docker container (list
>>> of dependencies include PETSc and petsc4py).
>>> >> >>>> >>> >> >>
>>> >> >>>> >>> >> >> I'm pretty sure this is not intended behaviour. Any
>>> insight as to how to fix this issue (I tried running ./configure
>>> --with-hdf5 to no avail) or more generally to perform this jiggling between
>>> real and complex would be much appreciated,
>>> >> >>>> >>> >> >>
>>> >> >>>> >>> >> >> Kind regards.
>>> >> >>>> >>> >> >>
>>> >> >>>> >>> >> >> Quentin
>>> >> >>>> >>> >> >
>>> >> >>>> >>> >> >
>>> >> >>>> >>> >> >
>>> >> >>>> >>> >> > --
>>> >> >>>> >>> >> > What most experimenters take for granted before they
>>> begin their experiments is infinitely more interesting than any results to
>>> which their experiments lead.
>>> >> >>>> >>> >> > -- Norbert Wiener
>>> >> >>>> >>> >> >
>>> >> >>>> >>> >> > https://www.cse.buffalo.edu/~knepley/
>>> >> >>>> >>> >
>>> >> >>>> >>> >
>>> >> >>>> >>> >
>>> >> >>>> >>> > --
>>> >> >>>> >>> > What most experimenters take for granted before they begin
>>> their experiments is infinitely more interesting than any results to which
>>> their experiments lead.
>>> >> >>>> >>> > -- Norbert Wiener
>>> >> >>>> >>> >
>>> >> >>>> >>> > https://www.cse.buffalo.edu/~knepley/
>>> >> >>>> >>
>>> >> >>>> >>
>>> >> >>>> >>
>>> >> >>>> >> --
>>> >> >>>> >> What most experimenters take for granted before they begin
>>> their experiments is infinitely more interesting than any results to which
>>> their experiments lead.
>>> >> >>>> >> -- Norbert Wiener
>>> >> >>>> >>
>>> >> >>>> >> https://www.cse.buffalo.edu/~knepley/
>>> >> >>>> >
>>> >> >>>> > <image003.jpg>
>>> >> >>>> >
>>> >> >>>> >
>>> >> >>>
>>> >> >>>
>>> >> >>>
>>> >> >>> --
>>> >> >>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> >> >>> -- Norbert Wiener
>>> >> >>>
>>> >> >>> https://www.cse.buffalo.edu/~knepley/
>>> >> >
>>> >> >
>>> >> >
>>> >> > --
>>> >> > What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> >> > -- Norbert Wiener
>>> >> >
>>> >> > https://www.cse.buffalo.edu/~knepley/
>>> >
>>> >
>>> >
>>> > --
>>> > What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> > -- Norbert Wiener
>>> >
>>> > https://www.cse.buffalo.edu/~knepley/
>>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>> https://www.cse.buffalo.edu/~knepley/
>> <http://www.cse.buffalo.edu/~knepley/>
>>
>
--
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20211208/ed92bf52/attachment-0001.html>
More information about the petsc-users
mailing list