[petsc-users] How to read/write a HDF5 file using petsc4py ?

Quentin Chevalier quentin.chevalier at polytechnique.edu
Wed Dec 8 04:03:54 CST 2021 

For the record, I tried changing the dockerfile with the --with-hdf5
flag and rebuilding a docker image, this changes the traceback to :

HDF5-DIAG: Error detected in HDF5 (1.12.1) MPI-process 0:
  #000: H5F.c line 620 in H5Fopen(): unable to open file
    major: File accessibility
    minor: Unable to open file
  #001: H5VLcallback.c line 3501 in H5VL_file_open(): failed to
iterate over available VOL connector plugins
    major: Virtual Object Layer
    minor: Iteration failed
  #002: H5PLpath.c line 578 in H5PL__path_table_iterate(): can't
iterate over plugins in plugin path '(null)'
    major: Plugin for dynamically loaded library
    minor: Iteration failed
  #003: H5PLpath.c line 620 in
H5PL__path_table_iterate_process_path(): can't open directory:
/usr/local/hdf5/lib/plugin
    major: Plugin for dynamically loaded library
    minor: Can't open directory or file
  #004: H5VLcallback.c line 3351 in H5VL__file_open(): open failed
    major: Virtual Object Layer
    minor: Can't open object
  #005: H5VLnative_file.c line 97 in H5VL__native_file_open(): unable
to open file
    major: File accessibility
    minor: Unable to open file
  #006: H5Fint.c line 1834 in H5F_open(): unable to open file: name =
'dummy.h5', tent_flags = 0
    major: File accessibility
    minor: Unable to open file
  #007: H5FD.c line 723 in H5FD_open(): open failed
    major: Virtual File Layer
    minor: Unable to initialize object
  #008: H5FDmpio.c line 850 in H5FD__mpio_open(): MPI_File_open
failed: MPI error string is 'File does not exist, error stack:
ADIOI_UFS_OPEN(37): File dummy.h5 does not exist'
    major: Internal error (too specific to document in detail)
    minor: Some MPI function failed
HDF5-DIAG: Error detected in HDF5 (1.12.1) MPI-process 0:
  #000: H5F.c line 707 in H5Fclose(): not a file ID
    major: Invalid arguments to routine
    minor: Inappropriate type
petsc4py.PETSc.Error: error code 76
[0] PetscObjectDestroy() at /usr/local/petsc/src/sys/objects/destroy.c:59
[0] PetscViewerDestroy() at
/usr/local/petsc/src/sys/classes/viewer/interface/view.c:119
[0] PetscViewerDestroy_HDF5() at
/usr/local/petsc/src/sys/classes/viewer/impls/hdf5/hdf5v.c:93
[0] PetscViewerFileClose_HDF5() at
/usr/local/petsc/src/sys/classes/viewer/impls/hdf5/hdf5v.c:82
[0] Error in external library
[0] Error in HDF5 call H5Fclose() Status -1
Exception ignored in: 'petsc4py.PETSc.Object.__dealloc__'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
petsc4py.PETSc.Error: error code 76
[0] PetscObjectDestroy() at /usr/local/petsc/src/sys/objects/destroy.c:59
[0] PetscViewerDestroy() at
/usr/local/petsc/src/sys/classes/viewer/interface/view.c:119
[0] PetscViewerDestroy_HDF5() at
/usr/local/petsc/src/sys/classes/viewer/impls/hdf5/hdf5v.c:93
[0] PetscViewerFileClose_HDF5() at
/usr/local/petsc/src/sys/classes/viewer/impls/hdf5/hdf5v.c:82
[0] Error in external library
[0] Error in HDF5 call H5Fclose() Status -1
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "PETSc/Viewer.pyx", line 184, in petsc4py.PETSc.Viewer.createHDF5
petsc4py.PETSc.Error: error code 76
[0] PetscObjectDestroy() at /usr/local/petsc/src/sys/objects/destroy.c:59
[0] PetscViewerDestroy() at
/usr/local/petsc/src/sys/classes/viewer/interface/view.c:119
[0] PetscViewerDestroy_HDF5() at
/usr/local/petsc/src/sys/classes/viewer/impls/hdf5/hdf5v.c:93
[0] PetscViewerFileClose_HDF5() at
/usr/local/petsc/src/sys/classes/viewer/impls/hdf5/hdf5v.c:82
[0] Error in external library
[0] Error in HDF5 call H5Fclose() Status -1

Since I'm calling upon the routine to create the file, and I have root
permissions over the container, it's got me confused.

Quentin



Quentin CHEVALIER – IA parcours recherche

LadHyX - Ecole polytechnique

__________



On Wed, 8 Dec 2021 at 10:07, Quentin Chevalier
<quentin.chevalier at polytechnique.edu> wrote:
>
> @all thanks for your time it's heartening to see a lively community.
>
> @Barry I've restarted the container and grabbed the .log file directly after the docker magic. I've tried a make check, it unsurprisingly spews the same answer as before :
>
> Running check examples to verify correct installation
> Using PETSC_DIR=/usr/local/petsc and PETSC_ARCH=linux-gnu-real-32
> /usr/bin/bash: line 1: cd: src/snes/tutorials: No such file or directory
> /usr/bin/bash: line 1: cd: src/snes/tutorials: No such file or directory
> gmake[3]: *** No rule to make target 'testex19'.  Stop.
> gmake[2]: *** [makefile:155: check_build] Error 2
>
>
> @Matthew ok will do, but I think @Lawrence has already provided that answer. It's possible to change the dockerfile and recompute the dolfinx image with hdf5, only it is a time-consuming process.
>
> Quentin
>
>
>
> Quentin CHEVALIER – IA parcours recherche
>
> LadHyX - Ecole polytechnique
>
> __________
>
>
>
> On Tue, 7 Dec 2021 at 19:16, Matthew Knepley <knepley at gmail.com> wrote:
>>
>> On Tue, Dec 7, 2021 at 9:43 AM Quentin Chevalier <quentin.chevalier at polytechnique.edu> wrote:
>>>
>>> @Matthew, as stated before, error output is unchanged, i.e.the python
>>> command below produces the same traceback :
>>>
>>> # python3 -c "from petsc4py import PETSc; PETSc.Viewer().createHDF5('d.h5')"
>>> Traceback (most recent call last):
>>>   File "<string>", line 1, in <module>
>>>   File "PETSc/Viewer.pyx", line 182, in petsc4py.PETSc.Viewer.createHDF5
>>> petsc4py.PETSc.Error: error code 86
>>> [0] PetscViewerSetType() at
>>> /usr/local/petsc/src/sys/classes/viewer/interface/viewreg.c:442
>>> [0] Unknown type. Check for miss-spelling or missing package:
>>> https://petsc.org/release/install/install/#external-packages
>>> [0] Unknown PetscViewer type given: hdf5
>>
>>
>> The reason I wanted the output was that the C output shows the configure options that the PETSc library
>> was built with, However, Python seems to be eating this, so I cannot check.
>>
>> It seems like using this container is counter-productive. If it was built correctly, making these changes would be trivial.
>> Send mail to FEniCS (I am guessing Chris Richardson maintains this), and ask how they intend people to change these
>> options.
>>
>>   Thanks,
>>
>>      Matt.
>>
>>>
>>> @Wence that makes sense. I'd assumed that the original PETSc had been
>>> overwritten, and if the linking has gone wrong I'm surprised anything
>>> happens with petsc4py at all.
>>>
>>> Your tentative command gave :
>>>
>>> ERROR: Invalid requirement: '/usr/local/petsc/src/binding/petsc4py'
>>> Hint: It looks like a path. File
>>> '/usr/local/petsc/src/binding/petsc4py' does not exist.
>>>
>>> So I tested that global variables PETSC_ARCH & PETSC_DIR were correct
>>> then ran "pip install petsc4py" to restart petsc4py from scratch. This
>>> gives rise to a different error :
>>> # python3 -c "from petsc4py import PETSc"
>>> Traceback (most recent call last):
>>>   File "<string>", line 1, in <module>
>>>   File "/usr/local/lib/python3.9/dist-packages/petsc4py/PETSc.py",
>>> line 3, in <module>
>>>     PETSc = ImportPETSc(ARCH)
>>>   File "/usr/local/lib/python3.9/dist-packages/petsc4py/lib/__init__.py",
>>> line 29, in ImportPETSc
>>>     return Import('petsc4py', 'PETSc', path, arch)
>>>   File "/usr/local/lib/python3.9/dist-packages/petsc4py/lib/__init__.py",
>>> line 73, in Import
>>>     module = import_module(pkg, name, path, arch)
>>>   File "/usr/local/lib/python3.9/dist-packages/petsc4py/lib/__init__.py",
>>> line 58, in import_module
>>>     with f: return imp.load_module(fullname, f, fn, info)
>>>   File "/usr/lib/python3.9/imp.py", line 242, in load_module
>>>     return load_dynamic(name, filename, file)
>>>   File "/usr/lib/python3.9/imp.py", line 342, in load_dynamic
>>>     return _load(spec)
>>> ImportError: /usr/local/lib/python3.9/dist-packages/petsc4py/lib/linux-gnu-real-32/PETSc.cpython-39-x86_64-linux-gnu.so:
>>> undefined symbol: petscstack
>>>
>>> Not sure that it a step forward ; looks like petsc4py is broken now.
>>>
>>> Quentin
>>>
>>> On Tue, 7 Dec 2021 at 14:58, Matthew Knepley <knepley at gmail.com> wrote:
>>> >
>>> > On Tue, Dec 7, 2021 at 8:26 AM Quentin Chevalier <quentin.chevalier at polytechnique.edu> wrote:
>>> >>
>>> >> Ok my bad, that log corresponded to a tentative --download-hdf5. This
>>> >> log corresponds to the commands given above and has --with-hdf5 in its
>>> >> options.
>>> >
>>> >
>>> > Okay, this configure was successful and found HDF5
>>> >
>>> >>
>>> >> The whole process still results in the same error.
>>> >
>>> >
>>> > Now send me the complete error output with this PETSc.
>>> >
>>> >   Thanks,
>>> >
>>> >      Matt
>>> >
>>> >>
>>> >> Quentin
>>> >>
>>> >>
>>> >>
>>> >> Quentin CHEVALIER – IA parcours recherche
>>> >>
>>> >> LadHyX - Ecole polytechnique
>>> >>
>>> >> __________
>>> >>
>>> >>
>>> >>
>>> >> On Tue, 7 Dec 2021 at 13:59, Matthew Knepley <knepley at gmail.com> wrote:
>>> >> >
>>> >> > On Tue, Dec 7, 2021 at 3:55 AM Quentin Chevalier <quentin.chevalier at polytechnique.edu> wrote:
>>> >> >>
>>> >> >> Hello Matthew,
>>> >> >>
>>> >> >> That would indeed make sense.
>>> >> >>
>>> >> >> Full log is attached, I grepped hdf5 in there and didn't find anything alarming.
>>> >> >
>>> >> >
>>> >> > At the top of this log:
>>> >> >
>>> >> > Configure Options: --configModules=PETSc.Configure --optionsModule=config.compilerOptions PETSC_ARCH=linux-gnu-complex-64 --COPTFLAGS=-O2 --CXXOPTFLAGS=-O2 --FOPTFLAGS=-O2 --with-make-np=2 --with-64-bit-indices=yes --with-debugging=no --with-fortran-bindings=no --with-shared-libraries --download-hypre --download-mumps --download-ptscotch --download-scalapack --download-suitesparse --download-superlu_dist --with-scalar-type=complex
>>> >> >
>>> >> >
>>> >> > So the HDF5 option is not being specified.
>>> >> >
>>> >> >   Thanks,
>>> >> >
>>> >> >      Matt
>>> >> >
>>> >> >> Cheers,
>>> >> >>
>>> >> >> Quentin
>>> >> >>
>>> >> >>
>>> >> >>
>>> >> >>
>>> >> >> Quentin CHEVALIER – IA parcours recherche
>>> >> >>
>>> >> >> LadHyX - Ecole polytechnique
>>> >> >>
>>> >> >> __________
>>> >> >>
>>> >> >>
>>> >> >>
>>> >> >> On Mon, 6 Dec 2021 at 21:39, Matthew Knepley <knepley at gmail.com> wrote:
>>> >> >>>
>>> >> >>> On Mon, Dec 6, 2021 at 3:27 PM Quentin Chevalier <quentin.chevalier at polytechnique.edu> wrote:
>>> >> >>>>
>>> >> >>>> Fine. MWE is unchanged :
>>> >> >>>> * Run this docker container
>>> >> >>>> * Do : python3 -c "from petsc4py import PETSc; PETSc.Viewer().createHDF5('dummy.h5')"
>>> >> >>>>
>>> >> >>>> Updated attempt at a fix :
>>> >> >>>> * cd /usr/local/petsc/
>>> >> >>>> * ./configure PETSC_ARCH= linux-gnu-real-32  PETSC_DIR=/usr/local/petsc --with-hdf5 --force
>>> >> >>>
>>> >> >>>
>>> >> >>> Did it find HDF5? If not, it will shut it off. You need to send us
>>> >> >>>
>>> >> >>>   $PETSC_DIR/configure.log
>>> >> >>>
>>> >> >>> so we can see what happened in the configure run.
>>> >> >>>
>>> >> >>>   Thanks,
>>> >> >>>
>>> >> >>>       Matt
>>> >> >>>
>>> >> >>>>
>>> >> >>>> * make PETSC_DIR=/usr/local/petsc PETSC-ARCH= linux-gnu-real-32 all
>>> >> >>>>
>>> >> >>>> Still no joy. The same error remains.
>>> >> >>>>
>>> >> >>>> Quentin
>>> >> >>>>
>>> >> >>>>
>>> >> >>>>
>>> >> >>>> On Mon, 6 Dec 2021 at 20:04, Pierre Jolivet <pierre at joliv.et> wrote:
>>> >> >>>> >
>>> >> >>>> >
>>> >> >>>> >
>>> >> >>>> > On 6 Dec 2021, at 7:42 PM, Quentin Chevalier <quentin.chevalier at polytechnique.edu> wrote:
>>> >> >>>> >
>>> >> >>>> > The PETSC_DIR exactly corresponds to the previous one, so I guess that rules option b) out, except if a specific option is required to overwrite a previous installation of PETSc. As for a), well I thought reconfiguring pretty direct, you're welcome to give me a hint as to what could be wrong in the following process.
>>> >> >>>> >
>>> >> >>>> > Steps to reproduce this behaviour are as follows :
>>> >> >>>> > * Run this docker container
>>> >> >>>> > * Do : python3 -c "from petsc4py import PETSc; PETSc.Viewer().createHDF5('dummy.h5')"
>>> >> >>>> >
>>> >> >>>> > After you get the error Unknown PetscViewer type, feel free to try :
>>> >> >>>> >
>>> >> >>>> > * cd /usr/local/petsc/
>>> >> >>>> > * ./configure --with-hfd5
>>> >> >>>> >
>>> >> >>>> >
>>> >> >>>> > It’s hdf5, not hfd5.
>>> >> >>>> > It’s PETSC_ARCH, not PETSC-ARCH.
>>> >> >>>> > PETSC_ARCH is missing from your configure line.
>>> >> >>>> >
>>> >> >>>> > Thanks,
>>> >> >>>> > Pierre
>>> >> >>>> >
>>> >> >>>> > * make PETSC_DIR=/usr/local/petsc PETSC-ARCH=linux-gnu-real-32 all
>>> >> >>>> >
>>> >> >>>> > Then repeat the MWE and observe absolutely no behavioural change whatsoever. I'm afraid I don't know PETSc well enough to be surprised by that.
>>> >> >>>> >
>>> >> >>>> > Quentin
>>> >> >>>> >
>>> >> >>>> >
>>> >> >>>> >
>>> >> >>>> > Quentin CHEVALIER – IA parcours recherche
>>> >> >>>> >
>>> >> >>>> > LadHyX - Ecole polytechnique
>>> >> >>>> >
>>> >> >>>> > __________
>>> >> >>>> >
>>> >> >>>> >
>>> >> >>>> >
>>> >> >>>> > On Mon, 6 Dec 2021 at 19:24, Matthew Knepley <knepley at gmail.com> wrote:
>>> >> >>>> >>
>>> >> >>>> >> On Mon, Dec 6, 2021 at 1:22 PM Quentin Chevalier <quentin.chevalier at polytechnique.edu> wrote:
>>> >> >>>> >>>
>>> >> >>>> >>> It failed all of the tests included in `make
>>> >> >>>> >>> PETSC_DIR=/usr/local/petsc PETSC-ARCH=linux-gnu-real-32 check`, with
>>> >> >>>> >>> the error `/usr/bin/bash: line 1: cd: src/snes/tutorials: No such file
>>> >> >>>> >>> or directory`
>>> >> >>>> >>>
>>> >> >>>> >>> I am therefore fairly confident this a "file absence" problem, and not
>>> >> >>>> >>> a compilation problem.
>>> >> >>>> >>>
>>> >> >>>> >>> I repeat that there was no error at compilation stage. The final stage
>>> >> >>>> >>> did present `gmake[3]: Nothing to be done for 'libs'.` but that's all.
>>> >> >>>> >>>
>>> >> >>>> >>> Again, running `./configure --with-hdf5` followed by a `make
>>> >> >>>> >>> PETSC_DIR=/usr/local/petsc PETSC-ARCH=linux-gnu-real-32 all` does not
>>> >> >>>> >>> change the problem. I get the same error at the same position as
>>> >> >>>> >>> before.
>>> >> >>>> >>
>>> >> >>>> >>
>>> >> >>>> >> If you reconfigured and rebuilt, it is impossible to get the same error, so
>>> >> >>>> >>
>>> >> >>>> >>   a) You did not reconfigure
>>> >> >>>> >>
>>> >> >>>> >>   b) Your new build is somewhere else on the machine
>>> >> >>>> >>
>>> >> >>>> >>   Thanks,
>>> >> >>>> >>
>>> >> >>>> >>      Matt
>>> >> >>>> >>
>>> >> >>>> >>>
>>> >> >>>> >>> I will comment I am running on OpenSUSE.
>>> >> >>>> >>>
>>> >> >>>> >>> Quentin
>>> >> >>>> >>>
>>> >> >>>> >>> On Mon, 6 Dec 2021 at 19:09, Matthew Knepley <knepley at gmail.com> wrote:
>>> >> >>>> >>> >
>>> >> >>>> >>> > On Mon, Dec 6, 2021 at 1:08 PM Quentin Chevalier <quentin.chevalier at polytechnique.edu> wrote:
>>> >> >>>> >>> >>
>>> >> >>>> >>> >> Hello Matthew and thanks for your quick response.
>>> >> >>>> >>> >>
>>> >> >>>> >>> >> I'm afraid I did try to snoop around the container and rerun PETSc's
>>> >> >>>> >>> >> configure with the --with-hdf5 option, to absolutely no avail.
>>> >> >>>> >>> >>
>>> >> >>>> >>> >> I didn't see any errors during config or make, but it failed the tests
>>> >> >>>> >>> >> (which aren't included in the minimal container I suppose)
>>> >> >>>> >>> >
>>> >> >>>> >>> >
>>> >> >>>> >>> > Failed which tests? What was the error?
>>> >> >>>> >>> >
>>> >> >>>> >>> >   Thanks,
>>> >> >>>> >>> >
>>> >> >>>> >>> >     Matt
>>> >> >>>> >>> >
>>> >> >>>> >>> >>
>>> >> >>>> >>> >> Quentin
>>> >> >>>> >>> >>
>>> >> >>>> >>> >>
>>> >> >>>> >>> >>
>>> >> >>>> >>> >> Quentin CHEVALIER – IA parcours recherche
>>> >> >>>> >>> >>
>>> >> >>>> >>> >> LadHyX - Ecole polytechnique
>>> >> >>>> >>> >>
>>> >> >>>> >>> >> __________
>>> >> >>>> >>> >>
>>> >> >>>> >>> >>
>>> >> >>>> >>> >>
>>> >> >>>> >>> >> On Mon, 6 Dec 2021 at 19:02, Matthew Knepley <knepley at gmail.com> wrote:
>>> >> >>>> >>> >> >
>>> >> >>>> >>> >> > On Mon, Dec 6, 2021 at 11:28 AM Quentin Chevalier <quentin.chevalier at polytechnique.edu> wrote:
>>> >> >>>> >>> >> >>
>>> >> >>>> >>> >> >> Hello PETSc users,
>>> >> >>>> >>> >> >>
>>> >> >>>> >>> >> >> This email is a duplicata of this gitlab issue, sorry for any inconvenience caused.
>>> >> >>>> >>> >> >>
>>> >> >>>> >>> >> >> I want to compute a PETSc vector in real mode, than perform calculations with it in complex mode. I want as much of this process to be parallel as possible. Right now, I compile PETSc in real mode, compute my vector and save it to a file, then switch to complex mode, read it, and move on.
>>> >> >>>> >>> >> >>
>>> >> >>>> >>> >> >> This creates unexpected behaviour using MPIIO, so on Lisandro Dalcinl's advice I'm moving to HDF5 format. My code is as follows (taking inspiration from petsc4py doc, a bitbucket example and another one, all top Google results for 'petsc hdf5') :
>>> >> >>>> >>> >> >>>
>>> >> >>>> >>> >> >>> viewer = PETSc.Viewer().createHDF5(file_name, 'r', COMM_WORLD)
>>> >> >>>> >>> >> >>> q.load(viewer)
>>> >> >>>> >>> >> >>> q.ghostUpdate(addv=PETSc.InsertMode.INSERT, mode=PETSc.ScatterMode.FORWARD)
>>> >> >>>> >>> >> >>
>>> >> >>>> >>> >> >>
>>> >> >>>> >>> >> >> This crashes my code. I obtain traceback :
>>> >> >>>> >>> >> >>>
>>> >> >>>> >>> >> >>>   File "/home/shared/code.py", line 121, in Load
>>> >> >>>> >>> >> >>>     viewer = PETSc.Viewer().createHDF5(file_name, 'r', COMM_WORLD)
>>> >> >>>> >>> >> >>>   File "PETSc/Viewer.pyx", line 182, in petsc4py.PETSc.Viewer.createHDF5
>>> >> >>>> >>> >> >>> petsc4py.PETSc.Error: error code 86
>>> >> >>>> >>> >> >>> [0] PetscViewerSetType() at /usr/local/petsc/src/sys/classes/viewer/interface/viewreg.c:442
>>> >> >>>> >>> >> >>> [0] Unknown type. Check for miss-spelling or missing package: https://petsc.org/release/install/install/#external-packages
>>> >> >>>> >>> >> >>> [0] Unknown PetscViewer type given: hdf5
>>> >> >>>> >>> >> >
>>> >> >>>> >>> >> > This means that PETSc has not been configured with HDF5 (--with-hdf5 or --download-hdf5), so the container should be updated.
>>> >> >>>> >>> >> >
>>> >> >>>> >>> >> >   THanks,
>>> >> >>>> >>> >> >
>>> >> >>>> >>> >> >     Matt
>>> >> >>>> >>> >> >
>>> >> >>>> >>> >> >>
>>> >> >>>> >>> >> >> I have petsc4py 3.16 from this docker container (list of dependencies include PETSc and petsc4py).
>>> >> >>>> >>> >> >>
>>> >> >>>> >>> >> >> I'm pretty sure this is not intended behaviour. Any insight as to how to fix this issue (I tried running ./configure --with-hdf5 to no avail) or more generally to perform this jiggling between real and complex would be much appreciated,
>>> >> >>>> >>> >> >>
>>> >> >>>> >>> >> >> Kind regards.
>>> >> >>>> >>> >> >>
>>> >> >>>> >>> >> >> Quentin
>>> >> >>>> >>> >> >
>>> >> >>>> >>> >> >
>>> >> >>>> >>> >> >
>>> >> >>>> >>> >> > --
>>> >> >>>> >>> >> > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>>> >> >>>> >>> >> > -- Norbert Wiener
>>> >> >>>> >>> >> >
>>> >> >>>> >>> >> > https://www.cse.buffalo.edu/~knepley/
>>> >> >>>> >>> >
>>> >> >>>> >>> >
>>> >> >>>> >>> >
>>> >> >>>> >>> > --
>>> >> >>>> >>> > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>>> >> >>>> >>> > -- Norbert Wiener
>>> >> >>>> >>> >
>>> >> >>>> >>> > https://www.cse.buffalo.edu/~knepley/
>>> >> >>>> >>
>>> >> >>>> >>
>>> >> >>>> >>
>>> >> >>>> >> --
>>> >> >>>> >> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>>> >> >>>> >> -- Norbert Wiener
>>> >> >>>> >>
>>> >> >>>> >> https://www.cse.buffalo.edu/~knepley/
>>> >> >>>> >
>>> >> >>>> > <image003.jpg>
>>> >> >>>> >
>>> >> >>>> >
>>> >> >>>
>>> >> >>>
>>> >> >>>
>>> >> >>> --
>>> >> >>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>>> >> >>> -- Norbert Wiener
>>> >> >>>
>>> >> >>> https://www.cse.buffalo.edu/~knepley/
>>> >> >
>>> >> >
>>> >> >
>>> >> > --
>>> >> > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>>> >> > -- Norbert Wiener
>>> >> >
>>> >> > https://www.cse.buffalo.edu/~knepley/
>>> >
>>> >
>>> >
>>> > --
>>> > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>>> > -- Norbert Wiener
>>> >
>>> > https://www.cse.buffalo.edu/~knepley/
>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>> -- Norbert Wiener
>>
>> https://www.cse.buffalo.edu/~knepley/

More information about the petsc-users mailing list